Nov 02 17:37:18.801926 ip-10-0-135-9 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 17:37:18.801935 ip-10-0-135-9 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 17:37:18.801942 ip-10-0-135-9 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 17:37:18.802163 ip-10-0-135-9 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 17:37:29.051205 ip-10-0-135-9 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 17:37:29.051223 ip-10-0-135-9 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e49811611f49406db19ddad229db1435 -- Nov 02 17:39:22.384372 ip-10-0-135-9 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:39:22.800240 ip-10-0-135-9 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:39:22.800240 ip-10-0-135-9 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:39:22.800240 ip-10-0-135-9 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:39:22.800240 ip-10-0-135-9 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:39:22.809837 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809740 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:22.809837 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809833 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:22.809837 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809837 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:22.809837 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809840 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:22.809837 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809843 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:22.809837 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809846 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809849 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809852 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809855 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809857 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809860 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809863 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809865 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809868 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809870 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809873 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809876 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809878 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809881 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809884 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809886 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809889 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809892 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809894 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:22.810067 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809897 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809900 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809902 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809905 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809907 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809910 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809912 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809915 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809918 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809921 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809923 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809926 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809928 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809931 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809934 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809937 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809940 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809943 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809945 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:22.810540 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809948 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809951 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809954 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809956 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809959 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809962 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809965 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809967 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809970 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809972 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809975 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809977 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809980 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809983 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809985 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809988 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809990 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809993 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.809996 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:22.810986 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810000 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810004 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810007 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810010 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810013 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810016 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810018 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810021 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810024 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810028 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810031 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810034 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810036 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810039 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810042 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810044 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810047 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810049 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810052 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810054 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:22.811435 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810057 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810067 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810070 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810073 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810091 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810094 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810097 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810101 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810104 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810107 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810109 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810112 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810115 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810118 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810121 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810123 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810127 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810131 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810134 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810137 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:22.811910 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810139 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810144 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810147 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810150 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810992 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.810999 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811002 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811005 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811008 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811011 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811013 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811016 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811019 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811022 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811026 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811029 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811031 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811034 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811037 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:22.812479 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811040 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811042 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811045 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811048 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811051 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811053 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811057 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811060 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811062 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811065 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811067 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811070 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811089 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811094 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811098 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811101 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811103 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811106 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811109 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:22.812938 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811111 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811114 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811117 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811120 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811122 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811125 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811128 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811130 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811133 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811136 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811139 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811142 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811144 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811147 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811150 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811152 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811155 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811157 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811160 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811163 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:22.813413 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811166 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811169 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811171 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811174 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811176 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811180 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811183 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811186 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811188 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811191 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811194 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811196 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811199 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811202 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811205 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811207 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811210 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811213 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811216 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811218 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:22.813889 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811221 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811224 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811227 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811230 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811232 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811235 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811238 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811240 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811243 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811246 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811248 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811251 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811254 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811256 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811259 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811262 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811264 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811267 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811270 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:22.814382 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811273 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811275 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811279 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811283 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811286 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811288 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811291 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811294 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811296 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811299 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811301 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811305 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811307 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811809 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811814 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811818 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811821 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811824 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811827 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:22.814828 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811829 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811832 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811835 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811838 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811841 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811843 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811846 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811849 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811853 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811856 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811859 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811862 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811864 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811867 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811870 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811872 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811875 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811879 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811883 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:22.815291 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811886 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811889 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811891 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811894 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811896 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811899 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811902 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811904 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811907 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811910 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811914 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811917 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811920 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811923 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811926 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811929 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811931 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811934 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811937 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811939 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:22.815727 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811942 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811944 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811947 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811949 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811952 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811955 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811957 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811960 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811963 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811965 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811968 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811970 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811973 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811976 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811978 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811981 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811983 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811986 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811988 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:22.816218 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811991 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811993 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811996 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.811999 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812002 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812005 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812008 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812011 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812013 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812016 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812018 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812021 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812024 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812026 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812029 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812032 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812034 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812038 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812041 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812043 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812046 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:22.816690 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812048 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812051 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812053 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812056 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812058 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812061 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812063 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812066 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812068 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812071 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812086 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812089 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812092 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812095 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812107 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812109 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812113 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812115 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812118 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812121 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:22.817247 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812123 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812214 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812222 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812228 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812234 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812240 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812244 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812251 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812257 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812262 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812265 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812268 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812272 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812275 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812278 2580 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812281 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812284 2580 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812296 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812299 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812303 2580 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812306 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812309 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812312 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812316 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:39:22.817727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812323 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812326 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812332 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812335 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812339 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812343 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812346 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812349 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812352 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812356 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812359 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812362 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812365 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812368 2580 flags.go:64] FLAG: --enable-server="true" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812371 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812377 2580 flags.go:64] FLAG: --event-burst="100" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812380 2580 flags.go:64] FLAG: --event-qps="50" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812383 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812386 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812389 2580 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812394 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812397 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812400 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812403 2580 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812406 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:39:22.818388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812409 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812412 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812415 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812418 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812420 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812423 2580 flags.go:64] FLAG: --feature-gates="" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812428 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812431 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812434 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812438 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812441 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812445 2580 flags.go:64] FLAG: --help="false" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812447 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812451 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812456 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812459 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812463 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812466 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812469 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812472 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812475 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812478 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812481 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:39:22.819034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812484 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812487 2580 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812490 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812494 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812497 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812500 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812503 2580 flags.go:64] FLAG: --lock-file="" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812506 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812509 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812512 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812518 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812521 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812524 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812527 2580 flags.go:64] FLAG: --logging-format="text" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812530 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812534 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812537 2580 flags.go:64] FLAG: --manifest-url="" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812540 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812545 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812549 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812552 2580 flags.go:64] FLAG: --max-pods="110" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812555 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812558 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812563 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812567 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:39:22.819607 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812570 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812572 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812575 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812583 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812586 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812589 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812592 2580 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812595 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812598 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812601 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812604 2580 flags.go:64] FLAG: --port="10250" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812607 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812610 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ab4fbfc104dbe07a" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812613 2580 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812616 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812619 2580 flags.go:64] FLAG: --register-node="true" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812622 2580 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812626 2580 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812629 2580 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812632 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812635 2580 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812639 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812642 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812645 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812648 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812653 2580 flags.go:64] FLAG: --runonce="false" Nov 02 17:39:22.820254 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812656 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812659 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812662 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812665 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812669 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812673 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812676 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812679 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812682 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812685 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812688 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812692 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812695 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812698 2580 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812701 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812706 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812709 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812712 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812716 2580 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812718 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812721 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812724 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812727 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812730 2580 flags.go:64] FLAG: --v="2" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812735 2580 flags.go:64] FLAG: --version="false" Nov 02 17:39:22.820857 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812739 2580 flags.go:64] FLAG: --vmodule="" Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812743 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.812746 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812878 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812882 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812885 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812889 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812893 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812895 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812898 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812900 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812904 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812907 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812910 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812913 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812915 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812918 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812921 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812924 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:22.821461 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812927 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812930 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812932 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812935 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812938 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812941 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812944 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812946 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812949 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812952 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812954 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812957 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812960 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812962 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812965 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812967 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812970 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812972 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812976 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812981 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:22.821919 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812984 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812987 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812989 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812992 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812995 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.812998 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813001 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813004 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813007 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813009 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813012 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813015 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813018 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813020 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813023 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813026 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813028 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813031 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813034 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:22.822405 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813037 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813039 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813042 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813044 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813047 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813049 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813052 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813055 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813057 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813060 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813063 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813065 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813069 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813072 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813087 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813090 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813093 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813097 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813099 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813102 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:22.822866 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813105 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813108 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813111 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813114 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813116 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813119 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813121 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813124 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813126 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813129 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813132 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813134 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813137 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813140 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813143 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813147 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813151 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813154 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813156 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:22.823389 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813159 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813162 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813164 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813167 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813169 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813173 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813176 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813178 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813181 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813183 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813187 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813190 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.813217 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813253 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813257 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813265 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:22.823832 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813268 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813271 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813274 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813276 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813279 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813282 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813284 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813287 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813290 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813292 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813295 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813298 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813300 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813303 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813306 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813308 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813311 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813314 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813318 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:22.824217 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813322 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813325 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813329 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813332 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813334 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813337 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813340 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813342 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813346 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813349 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813352 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813354 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813357 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813360 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813363 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813366 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813369 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813371 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813374 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:22.824672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813377 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813379 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813382 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813385 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813388 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813390 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813392 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813395 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813397 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813400 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813403 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813405 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813408 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813411 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813413 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813417 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813420 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813423 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813426 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813428 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:22.825126 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813431 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813435 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813437 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813440 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813443 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813445 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813449 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813452 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813454 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813457 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813461 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813465 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813468 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813471 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813474 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813477 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813479 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813482 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813485 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:22.825625 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813488 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813490 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813493 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813496 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813498 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813501 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813504 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813506 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813510 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813513 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813515 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813518 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813521 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813523 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813527 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813530 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813532 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813535 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813537 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813540 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:22.826066 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813543 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813546 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813548 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813552 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813554 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.813557 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.820627 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.820641 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.820654 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.820659 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820716 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820722 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820726 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820730 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820734 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820738 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820741 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820744 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820747 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:22.826558 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820749 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820752 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820755 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820758 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820761 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820764 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820767 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820769 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820772 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820775 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820777 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820780 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820783 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820786 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820789 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820792 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820795 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820797 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820800 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820802 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:22.827009 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820806 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820809 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820811 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820814 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820817 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820820 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820822 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820825 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820828 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820830 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820833 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820836 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820839 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820842 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820844 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820847 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820850 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820852 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820855 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820857 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:22.827601 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820860 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820863 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820866 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820869 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820872 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820875 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820877 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820880 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820883 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820886 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820889 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820891 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820895 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820898 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820900 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820903 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820906 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820908 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820911 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820914 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:22.828833 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820916 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820919 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820922 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820924 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820927 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820930 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820932 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820935 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820937 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820940 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820943 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820946 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820948 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820951 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820953 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820956 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820959 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820963 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820966 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:22.829312 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820969 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820972 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820975 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820978 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820980 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820984 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820986 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820989 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820992 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820994 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820997 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.820999 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821002 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821005 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821007 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821010 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821013 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:22.829740 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821015 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.821043 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821219 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821224 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821227 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821230 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821233 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821235 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821238 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821240 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821243 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821246 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821248 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821251 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821254 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821256 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:22.830150 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821259 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821262 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821265 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821268 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821271 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821274 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821276 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821279 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821282 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821284 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821287 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821290 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821292 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821295 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821297 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821300 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821303 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821305 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821308 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:22.830538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821311 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821313 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821316 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821319 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821321 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821324 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821327 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821329 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821332 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821334 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821337 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821339 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821342 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821345 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821347 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821352 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821356 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821359 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821362 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821365 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:22.831216 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821367 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821370 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821373 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821375 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821377 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821380 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821383 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821385 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821388 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821390 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821393 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821395 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821398 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821400 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821403 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821405 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821408 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821411 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821413 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:22.831958 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821416 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821419 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821421 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821424 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821427 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821429 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821432 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821434 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821437 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821440 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821442 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821445 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821448 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821450 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821453 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821455 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821458 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821460 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821463 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:22.832672 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821467 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821471 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821475 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821478 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821481 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821484 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821487 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821490 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821492 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821495 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821497 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821500 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821503 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821505 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821508 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.821531 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:39:22.833158 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821564 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821567 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821570 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821573 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821576 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821579 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821582 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821585 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821588 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821590 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821593 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821596 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821598 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821601 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821603 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821606 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821609 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821611 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821614 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821616 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:39:22.833538 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821619 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821621 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821624 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821626 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821629 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821631 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821634 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821636 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821639 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821642 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821645 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821647 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821650 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821652 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821655 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821657 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821660 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821663 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821665 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821668 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:39:22.834024 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821671 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821673 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821676 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821678 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821681 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821684 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821686 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821689 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821691 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821694 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821696 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821699 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821702 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821705 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821709 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821715 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821718 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821721 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821723 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821726 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:39:22.834535 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821729 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821732 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821735 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821737 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821740 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821743 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821745 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821748 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821750 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821753 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821755 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821758 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821760 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821763 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821766 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821768 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821771 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821773 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821776 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:39:22.834999 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821779 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821781 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821784 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821786 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821789 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821791 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821794 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821796 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821800 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821803 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821806 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821808 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821811 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821813 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821816 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821818 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821821 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821825 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821828 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:39:22.835671 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821831 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821834 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821836 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821839 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821841 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821844 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821847 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:22.821851 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.821899 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.824311 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.825382 2580 server.go:1014] "Starting client certificate rotation" Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.825481 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.825510 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.827632 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:39:22.836189 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.830130 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:39:22.840655 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.840638 2580 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:39:22.844632 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.844616 2580 log.go:25] "Validated CRI v1 image API" Nov 02 17:39:22.845434 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.845407 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:39:22.849831 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.849812 2580 fs.go:135] Filesystem UUIDs: map[309965dd-6c80-4c76-9984-290f457a8012:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c7000704-6e98-4d53-b706-d32bccc558c0:/dev/nvme0n1p3] Nov 02 17:39:22.849873 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.849831 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 17:39:22.855163 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.855036 2580 manager.go:217] Machine: {Timestamp:2026-11-02 17:39:22.853557658 +0000 UTC m=+0.372067645 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099927 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23e029fb56654d512d7ab1b3dae068 SystemUUID:ec23e029-fb56-654d-512d-7ab1b3dae068 BootID:e4981161-1f49-406d-b19d-dad229db1435 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:93:e2:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:93:e2:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:97:af:ea:ed:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:39:22.855163 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.855155 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:39:22.855287 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.855232 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:39:22.858167 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.858144 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:39:22.858299 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.858170 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-9.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:39:22.858344 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.858308 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:39:22.858344 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.858316 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:39:22.858393 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.858358 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:39:22.858393 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.858378 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:39:22.859638 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.859627 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:39:22.863700 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.863686 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:39:22.864119 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.864107 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:39:22.864246 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.864236 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:39:22.864284 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.864267 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:39:22.864284 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.864277 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:39:22.864343 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.864286 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:39:22.864343 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.864296 2580 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:39:22.864343 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.864305 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:39:22.864638 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.864626 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:39:22.867122 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.867101 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:39:22.874578 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.874563 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:39:22.875754 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.875730 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:39:22.875855 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.875779 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:39:22.875987 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.875973 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:39:22.876046 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.876005 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:39:22.876046 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.876017 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:39:22.876046 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.876029 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:39:22.876046 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.876038 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:39:22.876046 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.876047 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:39:22.876661 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.876644 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:39:22.876730 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.876666 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:39:22.876730 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.876679 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:39:22.876730 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.876691 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:39:22.876730 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.876728 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:39:22.876922 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.876741 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:39:22.878218 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.878203 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:39:22.878312 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.878228 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:39:22.880269 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.880243 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 17:39:22.880348 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.880257 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-9.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 17:39:22.880668 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.880655 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-9.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 17:39:22.883283 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.883270 2580 server.go:1263] "Started kubelet" Nov 02 17:39:22.883385 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.883338 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:39:22.883437 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.883400 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:39:22.883483 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.883460 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:39:22.884041 ip-10-0-135-9 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:39:22.885866 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.885847 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:39:22.886684 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.886665 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:39:22.889345 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.889329 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 17:39:22.890275 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.889101 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-9.us-east-2.compute.internal.18e44b22702c98e5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 17:39:22.883242213 +0000 UTC m=+0.401752201,LastTimestamp:2026-11-02 17:39:22.883242213 +0000 UTC m=+0.401752201,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-9.us-east-2.compute.internal,}" Nov 02 17:39:22.892828 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.892809 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:39:22.892828 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.892820 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:39:22.892955 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.892876 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:22.893549 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.893529 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:39:22.893549 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.893551 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:39:22.893679 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.893529 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:39:22.893679 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.893619 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:39:22.893679 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.893629 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:39:22.893764 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.893684 2580 factory.go:55] Registering systemd factory Nov 02 17:39:22.893764 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.893696 2580 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:39:22.894143 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.894131 2580 factory.go:153] Registering CRI-O factory Nov 02 17:39:22.894201 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.894147 2580 factory.go:223] Registration of the crio container factory successfully Nov 02 17:39:22.894201 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.894196 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:39:22.894298 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.894212 2580 factory.go:103] Registering Raw factory Nov 02 17:39:22.894298 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.894227 2580 manager.go:1199] Started watching for new ooms in manager Nov 02 17:39:22.895285 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.895272 2580 manager.go:319] Starting recovery of all containers Nov 02 17:39:22.900986 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.900953 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:39:22.903707 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.903667 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-9.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 17:39:22.905555 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.905490 2580 manager.go:324] Recovery completed Nov 02 17:39:22.911382 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.911368 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:22.913053 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.913028 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:22.913133 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.913066 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:22.913133 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.913089 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:22.913615 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.913599 2580 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:39:22.913615 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.913612 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:39:22.913735 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.913624 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:22.913735 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.913631 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:39:22.914256 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.914243 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:22.914309 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.914267 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:22.914309 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.914277 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:22.915712 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.915698 2580 policy_none.go:50] "Start" Nov 02 17:39:22.915712 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.915714 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:39:22.915827 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.915724 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:39:22.917694 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.917681 2580 policy_none.go:44] "Start" Nov 02 17:39:22.918867 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.918797 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-9.us-east-2.compute.internal.18e44b2271f37847 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-9.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 17:39:22.913052743 +0000 UTC m=+0.431562731,LastTimestamp:2026-11-02 17:39:22.913052743 +0000 UTC m=+0.431562731,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-9.us-east-2.compute.internal,}" Nov 02 17:39:22.919054 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.919032 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 17:39:22.923591 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.923524 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-9.us-east-2.compute.internal.18e44b2271f3bf90 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-9.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 17:39:22.913070992 +0000 UTC m=+0.431580979,LastTimestamp:2026-11-02 17:39:22.913070992 +0000 UTC m=+0.431580979,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-9.us-east-2.compute.internal,}" Nov 02 17:39:22.933291 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.933230 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-9.us-east-2.compute.internal.18e44b2271f41998 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-9.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 17:39:22.91309404 +0000 UTC m=+0.431604031,LastTimestamp:2026-11-02 17:39:22.91309404 +0000 UTC m=+0.431604031,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-9.us-east-2.compute.internal,}" Nov 02 17:39:22.942720 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.942657 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-9.us-east-2.compute.internal.18e44b2271f37847\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-9.us-east-2.compute.internal.18e44b2271f37847 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-9.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 17:39:22.913052743 +0000 UTC m=+0.431562731,LastTimestamp:2026-11-02 17:39:22.914255613 +0000 UTC m=+0.432765601,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-9.us-east-2.compute.internal,}" Nov 02 17:39:22.945964 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.945947 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gddwk" Nov 02 17:39:22.956550 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.956486 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-9.us-east-2.compute.internal.18e44b2271f3bf90\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-9.us-east-2.compute.internal.18e44b2271f3bf90 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-9.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 17:39:22.913070992 +0000 UTC m=+0.431580979,LastTimestamp:2026-11-02 17:39:22.914271584 +0000 UTC m=+0.432781572,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-9.us-east-2.compute.internal,}" Nov 02 17:39:22.958391 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.958374 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gddwk" Nov 02 17:39:22.964698 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.964676 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:39:22.966031 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.965967 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-9.us-east-2.compute.internal.18e44b2271f41998\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-9.us-east-2.compute.internal.18e44b2271f41998 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-9.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 17:39:22.91309404 +0000 UTC m=+0.431604031,LastTimestamp:2026-11-02 17:39:22.914281046 +0000 UTC m=+0.432791034,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-9.us-east-2.compute.internal,}" Nov 02 17:39:22.967039 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.967021 2580 manager.go:341] "Starting Device Plugin manager" Nov 02 17:39:22.967141 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.967055 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:39:22.967141 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.967065 2580 server.go:87] "Starting device plugin registration server" Nov 02 17:39:22.967352 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.967341 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:39:22.967407 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.967353 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:39:22.967968 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.967515 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:39:22.967968 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.967597 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:39:22.967968 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:22.967604 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:39:22.967968 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.967952 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:39:22.968155 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:22.968004 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:23.081572 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.068116 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:23.081572 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.069380 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:23.081572 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.069419 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:23.081572 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.069434 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:23.081572 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.069470 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.081572 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.079272 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.081572 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.079293 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-9.us-east-2.compute.internal\": node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:23.081572 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.079710 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:39:23.081572 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.079734 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:39:23.081572 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.079756 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:39:23.081572 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.079795 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 17:39:23.093216 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.093198 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:39:23.180206 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.180011 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal"] Nov 02 17:39:23.180358 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.180266 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:23.181295 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.181276 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:23.181350 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.181311 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:23.181350 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.181325 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:23.183509 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.183496 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:23.183646 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.183632 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.183685 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.183661 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:23.184240 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.184224 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:23.184240 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.184235 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:23.184357 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.184256 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:23.184357 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.184268 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:23.184357 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.184258 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:23.184357 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.184343 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:23.186353 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.186340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.186414 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.186363 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:23.187030 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.187016 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:23.187103 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.187035 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:23.187103 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.187048 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:23.198679 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.198662 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:23.215552 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.215537 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" node="ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.218312 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.218292 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" node="ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.294753 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.294680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.294753 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.294758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.295038 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.294776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06cff5a60959fbf288446179900e7d21-config\") pod \"kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"06cff5a60959fbf288446179900e7d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.298837 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.298819 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:23.395586 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.395494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.395586 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.395530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.395586 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.395568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06cff5a60959fbf288446179900e7d21-config\") pod \"kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"06cff5a60959fbf288446179900e7d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.395816 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.395601 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.395816 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.395620 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.395816 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.395629 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06cff5a60959fbf288446179900e7d21-config\") pod \"kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"06cff5a60959fbf288446179900e7d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.399602 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.399586 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:23.500089 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.500058 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:23.521227 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.521207 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.523379 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.523365 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:23.600767 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.600724 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:23.701338 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.701259 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:23.801600 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.801563 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:23.825128 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.825101 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 17:39:23.825268 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.825250 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:39:23.901691 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:23.901661 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:23.963870 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.963768 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:34:23 +0000 UTC" deadline="2028-05-24 13:15:16.795515886 +0000 UTC" Nov 02 17:39:23.963870 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:23.963811 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13651h35m52.831708231s" Nov 02 17:39:24.002606 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:24.002574 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-9.us-east-2.compute.internal\" not found" Nov 02 17:39:24.016766 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.016742 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:39:24.017841 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.017825 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:39:24.017930 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.017854 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:39:24.017930 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.017864 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:39:24.048222 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.048196 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:39:24.093831 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.093806 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:24.111780 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.111752 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:39:24.114288 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.114271 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:39:24.120466 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.120437 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:39:24.125871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.125853 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:39:24.271978 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:24.271939 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cff5a60959fbf288446179900e7d21.slice/crio-b9586f27793af460b023b7e5c1741f9b7cd0d229c3cd46cc5e42631b3bab2bec WatchSource:0}: Error finding container b9586f27793af460b023b7e5c1741f9b7cd0d229c3cd46cc5e42631b3bab2bec: Status 404 returned error can't find the container with id b9586f27793af460b023b7e5c1741f9b7cd0d229c3cd46cc5e42631b3bab2bec Nov 02 17:39:24.272454 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:24.272429 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8fb63764518311609bba29a4e6d416.slice/crio-b4b3bee0a5dafecb36831bd35a2f800a2cf529c89088c55c27e447ed184570b3 WatchSource:0}: Error finding container b4b3bee0a5dafecb36831bd35a2f800a2cf529c89088c55c27e447ed184570b3: Status 404 returned error can't find the container with id b4b3bee0a5dafecb36831bd35a2f800a2cf529c89088c55c27e447ed184570b3 Nov 02 17:39:24.277493 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.277470 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:39:24.491373 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.491338 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:39:24.806310 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.806279 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:39:24.867427 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.867392 2580 apiserver.go:52] "Watching apiserver" Nov 02 17:39:24.874619 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.874591 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:39:24.874998 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.874970 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-pbcbk","kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal","openshift-dns/node-resolver-59bfz","openshift-image-registry/node-ca-2crqk","openshift-multus/multus-additional-cni-plugins-l8c9g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-x4786","openshift-multus/multus-ksps4","openshift-ovn-kubernetes/ovnkube-node-s8hhd","kube-system/konnectivity-agent-586q5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw","openshift-cluster-node-tuning-operator/tuned-8gq7s","openshift-multus/network-metrics-daemon-wtm26","openshift-network-operator/iptables-alerter-z7lsw"] Nov 02 17:39:24.877542 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.877515 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:24.877650 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:24.877606 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:24.880364 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.880341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:24.880459 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:24.880424 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:24.882300 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.882253 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-59bfz" Nov 02 17:39:24.884815 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.884794 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:39:24.886723 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.886703 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:39:24.887079 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.887065 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:39:24.888645 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.888621 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:24.889295 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.889253 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:39:24.889401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.889361 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:39:24.889471 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.889434 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:39:24.890110 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.890052 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zqcnj\"" Nov 02 17:39:24.890110 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.890058 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzjcr\"" Nov 02 17:39:24.890699 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.890681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:39:24.894203 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.893695 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 17:39:24.894203 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.893959 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.896475 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.896373 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:24.896475 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:24.896439 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:24.898547 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.898531 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:39:24.899303 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.899277 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ksps4" Nov 02 17:39:24.903561 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.903194 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:24.903934 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.903910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4307ecfe-be36-4891-9c9d-4c649e9aca70-tmp-dir\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:39:24.904035 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.903947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-serviceca\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:39:24.904035 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.903973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgpc\" (UniqueName: \"kubernetes.io/projected/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-kube-api-access-bwgpc\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:39:24.904035 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-system-cni-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:24.904035 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:24.904331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kln72\" (UniqueName: \"kubernetes.io/projected/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-kube-api-access-kln72\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:24.904331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-kubelet\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.904331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904111 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-netns\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.904331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rtjz\" (UniqueName: \"kubernetes.io/projected/4307ecfe-be36-4891-9c9d-4c649e9aca70-kube-api-access-8rtjz\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:39:24.904331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cnibin\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:24.904331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-binary-copy\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:24.904331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-log-socket\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.904331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d417b861-3a8a-4889-8a97-3a03132e8623-ovn-node-metrics-cert\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.904331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6nsz\" (UniqueName: \"kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz\") pod \"network-check-target-x4786\" (UID: \"95cc5486-946f-4ada-be5d-84fc0068d031\") " pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:24.904331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904291 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-host\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:39:24.904331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcqc6\" (UniqueName: \"kubernetes.io/projected/d417b861-3a8a-4889-8a97-3a03132e8623-kube-api-access-kcqc6\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904395 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61d00064-21c5-44b2-a467-18d93e752363-host-slash\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swdz\" (UniqueName: \"kubernetes.io/projected/61d00064-21c5-44b2-a467-18d93e752363-kube-api-access-9swdz\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-var-lib-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-node-log\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904506 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-systemd-units\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904545 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904780 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61d00064-21c5-44b2-a467-18d93e752363-iptables-alerter-script\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4307ecfe-be36-4891-9c9d-4c649e9aca70-hosts-file\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-etc-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.904943 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.904872 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.905694 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.905035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-env-overrides\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.905694 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.905073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-script-lib\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.905694 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.905108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8pkt\" (UniqueName: \"kubernetes.io/projected/9edee41d-f2cc-404e-ae90-33dd5312d523-kube-api-access-r8pkt\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:24.905694 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.905138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-slash\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.905694 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.905355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-systemd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.905694 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.905378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-ovn\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.905694 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.905404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-bin\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.905694 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.905429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-netd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.905694 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.905519 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-config\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:24.906211 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.906194 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:24.911804 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.911779 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:39:24.912839 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.912816 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:24.990134 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.990103 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cq7h2" Nov 02 17:39:24.994614 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.994588 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:39:24.995925 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.995880 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:39:24.996141 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.996122 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:39:24.996264 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.996195 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:39:24.996338 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.996266 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:39:24.996338 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.996327 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:39:24.996447 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.996364 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:39:24.996682 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.996663 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:39:24.996758 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.996725 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:39:24.996822 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.996798 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:39:24.996881 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.996854 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:39:24.996964 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.996949 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:39:24.997053 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.997040 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:39:24.997113 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.997082 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:39:24.997200 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.997182 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:39:24.997270 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.997209 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:39:24.997849 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.997830 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:39:24.997931 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.997879 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:39:24.998017 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.997991 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:39:24.998966 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.998915 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:39:24.999103 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.999068 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:39:24.999184 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.999174 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:39:24.999255 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.999182 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4zvsl\"" Nov 02 17:39:24.999255 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.999205 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-kkqh6\"" Nov 02 17:39:24.999255 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.999223 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rjwn5\"" Nov 02 17:39:24.999415 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.999278 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpgl9\"" Nov 02 17:39:24.999415 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.999290 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b2827\"" Nov 02 17:39:24.999415 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.999307 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4dl6t\"" Nov 02 17:39:24.999415 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:24.999366 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jk79z\"" Nov 02 17:39:25.001331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.001307 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cq7h2" Nov 02 17:39:25.006641 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-config\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.006752 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgpc\" (UniqueName: \"kubernetes.io/projected/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-kube-api-access-bwgpc\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:39:25.006752 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kln72\" (UniqueName: \"kubernetes.io/projected/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-kube-api-access-kln72\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.006752 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-konnectivity-ca\") pod \"konnectivity-agent-586q5\" (UID: \"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13\") " pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:25.006752 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.006966 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-run\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.006966 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cnibin\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.006966 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-host\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.006966 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-multus-certs\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.006966 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:25.006966 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.006966 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-multus\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.006966 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-kubelet-config\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-systemd-units\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.006999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-kubelet\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-kubernetes\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61d00064-21c5-44b2-a467-18d93e752363-iptables-alerter-script\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsrdf\" (UniqueName: \"kubernetes.io/projected/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-kube-api-access-qsrdf\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-agent-certs\") pod \"konnectivity-agent-586q5\" (UID: \"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13\") " pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-sys\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4307ecfe-be36-4891-9c9d-4c649e9aca70-hosts-file\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007159 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-etc-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-env-overrides\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-script-lib\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-etc-kubernetes\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007273 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-config\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-conf\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.007401 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-lib-modules\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.008116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8pkt\" (UniqueName: \"kubernetes.io/projected/9edee41d-f2cc-404e-ae90-33dd5312d523-kube-api-access-r8pkt\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:25.008116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-systemd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.008116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-bin\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.008116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007664 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cni-binary-copy\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.008116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-netns\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.008116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007784 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-bin\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.008116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007816 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cnibin\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.008116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007890 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4307ecfe-be36-4891-9c9d-4c649e9aca70-hosts-file\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:39:25.008116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007893 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-systemd-units\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.008116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.007921 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-etc-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.008502 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008379 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-script-lib\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.008502 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008424 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4307ecfe-be36-4891-9c9d-4c649e9aca70-tmp-dir\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:39:25.008502 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-serviceca\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:39:25.008502 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-system-cni-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.008502 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008472 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61d00064-21c5-44b2-a467-18d93e752363-iptables-alerter-script\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:39:25.008502 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.008779 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-kubelet\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.008779 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-netns\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.008779 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008575 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-bin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.008779 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008600 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-etc-selinux\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.008779 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rtjz\" (UniqueName: \"kubernetes.io/projected/4307ecfe-be36-4891-9c9d-4c649e9aca70-kube-api-access-8rtjz\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:39:25.008779 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-binary-copy\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.008779 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-log-socket\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.008779 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d417b861-3a8a-4889-8a97-3a03132e8623-ovn-node-metrics-cert\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.008779 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-socket-dir-parent\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.008779 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-daemon-config\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.008779 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nsz\" (UniqueName: \"kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz\") pod \"network-check-target-x4786\" (UID: \"95cc5486-946f-4ada-be5d-84fc0068d031\") " pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008803 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-env-overrides\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-host\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcqc6\" (UniqueName: \"kubernetes.io/projected/d417b861-3a8a-4889-8a97-3a03132e8623-kube-api-access-kcqc6\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-system-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008908 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cnibin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysconfig\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.008954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-dbus\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61d00064-21c5-44b2-a467-18d93e752363-host-slash\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swdz\" (UniqueName: \"kubernetes.io/projected/61d00064-21c5-44b2-a467-18d93e752363-kube-api-access-9swdz\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009065 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-var-lib-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-node-log\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.009304 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009134 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.009977 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009501 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.009977 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009580 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-systemd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.009977 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.009655 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:25.009977 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009689 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-log-socket\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.009977 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.009723 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs podName:9edee41d-f2cc-404e-ae90-33dd5312d523 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:25.509696867 +0000 UTC m=+3.028206858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs") pod "network-metrics-daemon-wtm26" (UID: "9edee41d-f2cc-404e-ae90-33dd5312d523") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:25.009977 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009849 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.010303 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.010082 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4307ecfe-be36-4891-9c9d-4c649e9aca70-tmp-dir\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:39:25.010303 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.010227 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-host\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:39:25.010473 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.010435 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:39:25.010600 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.010515 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61d00064-21c5-44b2-a467-18d93e752363-host-slash\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:39:25.010985 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.009656 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-binary-copy\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.011050 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.010979 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.011050 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-hostroot\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.011167 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011047 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-netns\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.011167 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbfg9\" (UniqueName: \"kubernetes.io/projected/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kube-api-access-fbfg9\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.011167 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011137 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-system-cni-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.011326 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-k8s-cni-cncf-io\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.011326 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-socket-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.011326 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.011326 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011307 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-systemd\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.011516 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011358 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-var-lib-kubelet\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.011516 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-tuned\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.011516 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.011516 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011412 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-kubelet\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.011516 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-sys-fs\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.011516 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-modprobe-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.011516 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-tmp\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.011862 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znq7v\" (UniqueName: \"kubernetes.io/projected/9762673b-765b-4d8c-8791-ee16e740c1b3-kube-api-access-znq7v\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.011862 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.011862 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-conf-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.011862 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-registration-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.011862 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-device-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.011862 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:25.011862 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-slash\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.011862 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-ovn\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.011862 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011808 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.011862 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-netd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.012364 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.011930 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-netd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.012364 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.012005 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-slash\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.012364 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.012007 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-node-log\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.012364 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.012042 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.012364 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.012074 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-var-lib-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.012364 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.012098 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-ovn\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.012656 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.012552 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-serviceca\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:39:25.015123 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.015098 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d417b861-3a8a-4889-8a97-3a03132e8623-ovn-node-metrics-cert\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.023373 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.023346 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:25.023501 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.023382 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:25.023501 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.023398 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q6nsz for pod openshift-network-diagnostics/network-check-target-x4786: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:25.023501 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.023474 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz podName:95cc5486-946f-4ada-be5d-84fc0068d031 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:25.523453083 +0000 UTC m=+3.041963078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q6nsz" (UniqueName: "kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz") pod "network-check-target-x4786" (UID: "95cc5486-946f-4ada-be5d-84fc0068d031") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:25.025679 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.025651 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rtjz\" (UniqueName: \"kubernetes.io/projected/4307ecfe-be36-4891-9c9d-4c649e9aca70-kube-api-access-8rtjz\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:39:25.025762 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.025650 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgpc\" (UniqueName: \"kubernetes.io/projected/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-kube-api-access-bwgpc\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:39:25.026406 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.026379 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8pkt\" (UniqueName: \"kubernetes.io/projected/9edee41d-f2cc-404e-ae90-33dd5312d523-kube-api-access-r8pkt\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:25.028208 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.028184 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kln72\" (UniqueName: \"kubernetes.io/projected/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-kube-api-access-kln72\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.028342 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.028320 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swdz\" (UniqueName: \"kubernetes.io/projected/61d00064-21c5-44b2-a467-18d93e752363-kube-api-access-9swdz\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:39:25.030855 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.030836 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcqc6\" (UniqueName: \"kubernetes.io/projected/d417b861-3a8a-4889-8a97-3a03132e8623-kube-api-access-kcqc6\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.085103 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.084974 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" event={"ID":"3b8fb63764518311609bba29a4e6d416","Type":"ContainerStarted","Data":"b4b3bee0a5dafecb36831bd35a2f800a2cf529c89088c55c27e447ed184570b3"} Nov 02 17:39:25.086583 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.086555 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" event={"ID":"06cff5a60959fbf288446179900e7d21","Type":"ContainerStarted","Data":"b9586f27793af460b023b7e5c1741f9b7cd0d229c3cd46cc5e42631b3bab2bec"} Nov 02 17:39:25.113185 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-registration-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.113366 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-device-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.113366 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:25.113366 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-konnectivity-ca\") pod \"konnectivity-agent-586q5\" (UID: \"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13\") " pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:25.113366 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.113366 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-run\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.113366 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-host\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.113366 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113358 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-multus-certs\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113593 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-multus\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113593 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-kubelet-config\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:25.113593 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-kubelet\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113593 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-kubernetes\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.113593 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrdf\" (UniqueName: \"kubernetes.io/projected/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-kube-api-access-qsrdf\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113593 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-agent-certs\") pod \"konnectivity-agent-586q5\" (UID: \"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13\") " pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:25.113593 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-sys\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.113593 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113593 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-etc-kubernetes\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113593 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-conf\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-lib-modules\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cni-binary-copy\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-netns\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-bin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-etc-selinux\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-socket-dir-parent\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-daemon-config\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-system-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cnibin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysconfig\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-dbus\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-hostroot\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbfg9\" (UniqueName: \"kubernetes.io/projected/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kube-api-access-fbfg9\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-k8s-cni-cncf-io\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-socket-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.113871 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-systemd\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-var-lib-kubelet\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-tuned\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-sys-fs\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-modprobe-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-tmp\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.113987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znq7v\" (UniqueName: \"kubernetes.io/projected/9762673b-765b-4d8c-8791-ee16e740c1b3-kube-api-access-znq7v\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.114004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-conf-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.114068 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-conf-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.114117 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-registration-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.114148 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-device-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.114256 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.114322 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret podName:ea8500af-fe54-4c1e-983d-7a7b82d866b3 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:25.614303987 +0000 UTC m=+3.132813969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret") pod "global-pull-secret-syncer-pbcbk" (UID: "ea8500af-fe54-4c1e-983d-7a7b82d866b3") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.115083 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-konnectivity-ca\") pod \"konnectivity-agent-586q5\" (UID: \"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13\") " pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:25.115221 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.115182 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.115849 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.115319 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-run\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.115849 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.115370 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-host\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.115849 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.115420 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-multus-certs\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.115849 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.115467 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-multus\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.115849 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.115520 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-kubelet-config\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:25.115849 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.115567 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-kubelet\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.115849 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.115618 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-kubernetes\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.116803 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.116783 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-sys\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.116968 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.116945 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.117052 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.117002 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-etc-kubernetes\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.117253 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.117190 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-conf\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.117350 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.117319 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-lib-modules\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.117824 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.117800 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-netns\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.117912 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.117861 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-bin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.117974 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.117940 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cni-binary-copy\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.117974 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.117965 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-etc-selinux\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.118084 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118023 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-socket-dir-parent\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.118084 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118064 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-k8s-cni-cncf-io\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.118217 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118082 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-socket-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.118296 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118278 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118348 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-systemd\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118435 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-var-lib-kubelet\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118451 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysconfig\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118523 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-system-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118576 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cnibin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118631 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-dbus\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118644 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-hostroot\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118710 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-sys-fs\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.118796 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-modprobe-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.119338 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-daemon-config\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.120280 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-agent-certs\") pod \"konnectivity-agent-586q5\" (UID: \"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13\") " pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.122840 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-tmp\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.130624 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.126707 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-tuned\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.136183 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.133064 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbfg9\" (UniqueName: \"kubernetes.io/projected/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kube-api-access-fbfg9\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.136183 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.133420 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrdf\" (UniqueName: \"kubernetes.io/projected/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-kube-api-access-qsrdf\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.136183 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.135099 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znq7v\" (UniqueName: \"kubernetes.io/projected/9762673b-765b-4d8c-8791-ee16e740c1b3-kube-api-access-znq7v\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.199198 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.199158 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-59bfz" Nov 02 17:39:25.207731 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.207704 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:39:25.210577 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:25.210152 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4307ecfe_be36_4891_9c9d_4c649e9aca70.slice/crio-65ff996d599a7608f7fe9d97fa4e09ee14011e646a4240e9c06654e0aa6f23db WatchSource:0}: Error finding container 65ff996d599a7608f7fe9d97fa4e09ee14011e646a4240e9c06654e0aa6f23db: Status 404 returned error can't find the container with id 65ff996d599a7608f7fe9d97fa4e09ee14011e646a4240e9c06654e0aa6f23db Nov 02 17:39:25.219643 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:25.219394 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b7e801_25bc_4f6e_be57_f3131d2ce39c.slice/crio-4d4854f8417a737d3de81577301ce23da7e44e0f890424ff66a81342462cfb98 WatchSource:0}: Error finding container 4d4854f8417a737d3de81577301ce23da7e44e0f890424ff66a81342462cfb98: Status 404 returned error can't find the container with id 4d4854f8417a737d3de81577301ce23da7e44e0f890424ff66a81342462cfb98 Nov 02 17:39:25.239939 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.239909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:39:25.240199 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.239920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:39:25.251965 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.251921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ksps4" Nov 02 17:39:25.253573 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:25.253544 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d00064_21c5_44b2_a467_18d93e752363.slice/crio-84ead3af633f8a0a68a58e26934899eb447ced88b7d61dc43a6a74e4d844f935 WatchSource:0}: Error finding container 84ead3af633f8a0a68a58e26934899eb447ced88b7d61dc43a6a74e4d844f935: Status 404 returned error can't find the container with id 84ead3af633f8a0a68a58e26934899eb447ced88b7d61dc43a6a74e4d844f935 Nov 02 17:39:25.258769 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:25.258731 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbe9efd_9dab_4adf_a72f_e5ce853fb19e.slice/crio-728f0b46110f27713f0ca1d3a806dc39fa67ac7702e9ee38f172c20af05a5264 WatchSource:0}: Error finding container 728f0b46110f27713f0ca1d3a806dc39fa67ac7702e9ee38f172c20af05a5264: Status 404 returned error can't find the container with id 728f0b46110f27713f0ca1d3a806dc39fa67ac7702e9ee38f172c20af05a5264 Nov 02 17:39:25.262108 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.262042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:25.264703 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.264683 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:25.264801 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:25.264779 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63aff75_00c5_4a5c_b9d3_678e6ed4465f.slice/crio-eb1271a6e0a5408d6d886fbeb865b4b9275b44bdb4ed32570506aa64b1920993 WatchSource:0}: Error finding container eb1271a6e0a5408d6d886fbeb865b4b9275b44bdb4ed32570506aa64b1920993: Status 404 returned error can't find the container with id eb1271a6e0a5408d6d886fbeb865b4b9275b44bdb4ed32570506aa64b1920993 Nov 02 17:39:25.274280 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.274253 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:39:25.274652 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:25.274630 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae0a868_357a_4fe8_a3d9_ea6aff6c4e13.slice/crio-53afcb4f0b6a07dad464bc1959e6a377e02b71721178be3604d11ec340930bd4 WatchSource:0}: Error finding container 53afcb4f0b6a07dad464bc1959e6a377e02b71721178be3604d11ec340930bd4: Status 404 returned error can't find the container with id 53afcb4f0b6a07dad464bc1959e6a377e02b71721178be3604d11ec340930bd4 Nov 02 17:39:25.276108 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.276087 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:39:25.279131 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:25.279032 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd417b861_3a8a_4889_8a97_3a03132e8623.slice/crio-128194c44a764a57564fb9be1145f552a6eb7f8d040141336abd66f6679d0a01 WatchSource:0}: Error finding container 128194c44a764a57564fb9be1145f552a6eb7f8d040141336abd66f6679d0a01: Status 404 returned error can't find the container with id 128194c44a764a57564fb9be1145f552a6eb7f8d040141336abd66f6679d0a01 Nov 02 17:39:25.288241 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:25.288211 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7cdb9f_dd7b_4b23_bf0a_0d23fb6e382e.slice/crio-e3d7515ca0e1a3c98d3379022398065f010623807d452a28d1550585dedfe092 WatchSource:0}: Error finding container e3d7515ca0e1a3c98d3379022398065f010623807d452a28d1550585dedfe092: Status 404 returned error can't find the container with id e3d7515ca0e1a3c98d3379022398065f010623807d452a28d1550585dedfe092 Nov 02 17:39:25.291703 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:25.291681 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9762673b_765b_4d8c_8791_ee16e740c1b3.slice/crio-652b0a3f0c4987e8b7b5648ba6dfd1bca6e25c41e197b9629f67bb37759901e3 WatchSource:0}: Error finding container 652b0a3f0c4987e8b7b5648ba6dfd1bca6e25c41e197b9629f67bb37759901e3: Status 404 returned error can't find the container with id 652b0a3f0c4987e8b7b5648ba6dfd1bca6e25c41e197b9629f67bb37759901e3 Nov 02 17:39:25.518034 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.517994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:25.518243 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.518187 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:25.518314 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.518266 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs podName:9edee41d-f2cc-404e-ae90-33dd5312d523 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:26.518245403 +0000 UTC m=+4.036755398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs") pod "network-metrics-daemon-wtm26" (UID: "9edee41d-f2cc-404e-ae90-33dd5312d523") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:25.619049 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.619009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:25.619255 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.619211 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:25.619309 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.619290 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret podName:ea8500af-fe54-4c1e-983d-7a7b82d866b3 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:26.619269602 +0000 UTC m=+4.137779595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret") pod "global-pull-secret-syncer-pbcbk" (UID: "ea8500af-fe54-4c1e-983d-7a7b82d866b3") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:25.619360 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:25.619332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nsz\" (UniqueName: \"kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz\") pod \"network-check-target-x4786\" (UID: \"95cc5486-946f-4ada-be5d-84fc0068d031\") " pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:25.619462 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.619443 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:25.619462 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.619460 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:25.619674 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.619472 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q6nsz for pod openshift-network-diagnostics/network-check-target-x4786: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:25.619674 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:25.619511 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz podName:95cc5486-946f-4ada-be5d-84fc0068d031 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:26.619499213 +0000 UTC m=+4.138009199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6nsz" (UniqueName: "kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz") pod "network-check-target-x4786" (UID: "95cc5486-946f-4ada-be5d-84fc0068d031") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:26.004767 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.004716 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:34:25 +0000 UTC" deadline="2028-05-25 01:32:32.724458146 +0000 UTC" Nov 02 17:39:26.004767 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.004765 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13663h53m6.719697584s" Nov 02 17:39:26.095358 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.095270 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ksps4" event={"ID":"b63aff75-00c5-4a5c-b9d3-678e6ed4465f","Type":"ContainerStarted","Data":"eb1271a6e0a5408d6d886fbeb865b4b9275b44bdb4ed32570506aa64b1920993"} Nov 02 17:39:26.097296 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.097262 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8c9g" event={"ID":"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e","Type":"ContainerStarted","Data":"728f0b46110f27713f0ca1d3a806dc39fa67ac7702e9ee38f172c20af05a5264"} Nov 02 17:39:26.107235 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.107196 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z7lsw" event={"ID":"61d00064-21c5-44b2-a467-18d93e752363","Type":"ContainerStarted","Data":"84ead3af633f8a0a68a58e26934899eb447ced88b7d61dc43a6a74e4d844f935"} Nov 02 17:39:26.117727 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.117696 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" event={"ID":"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e","Type":"ContainerStarted","Data":"e3d7515ca0e1a3c98d3379022398065f010623807d452a28d1550585dedfe092"} Nov 02 17:39:26.125713 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.125677 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2crqk" event={"ID":"a3b7e801-25bc-4f6e-be57-f3131d2ce39c","Type":"ContainerStarted","Data":"4d4854f8417a737d3de81577301ce23da7e44e0f890424ff66a81342462cfb98"} Nov 02 17:39:26.143332 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.143226 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-59bfz" event={"ID":"4307ecfe-be36-4891-9c9d-4c649e9aca70","Type":"ContainerStarted","Data":"65ff996d599a7608f7fe9d97fa4e09ee14011e646a4240e9c06654e0aa6f23db"} Nov 02 17:39:26.146555 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.146477 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" event={"ID":"9762673b-765b-4d8c-8791-ee16e740c1b3","Type":"ContainerStarted","Data":"652b0a3f0c4987e8b7b5648ba6dfd1bca6e25c41e197b9629f67bb37759901e3"} Nov 02 17:39:26.148517 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.148405 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" event={"ID":"d417b861-3a8a-4889-8a97-3a03132e8623","Type":"ContainerStarted","Data":"128194c44a764a57564fb9be1145f552a6eb7f8d040141336abd66f6679d0a01"} Nov 02 17:39:26.150591 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.150541 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-586q5" event={"ID":"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13","Type":"ContainerStarted","Data":"53afcb4f0b6a07dad464bc1959e6a377e02b71721178be3604d11ec340930bd4"} Nov 02 17:39:26.530375 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.530332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:26.530632 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:26.530501 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:26.530632 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:26.530574 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs podName:9edee41d-f2cc-404e-ae90-33dd5312d523 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:28.530555711 +0000 UTC m=+6.049065701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs") pod "network-metrics-daemon-wtm26" (UID: "9edee41d-f2cc-404e-ae90-33dd5312d523") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:26.631823 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.631764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nsz\" (UniqueName: \"kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz\") pod \"network-check-target-x4786\" (UID: \"95cc5486-946f-4ada-be5d-84fc0068d031\") " pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:26.632083 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:26.631843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:26.632083 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:26.632009 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:26.632083 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:26.632077 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret podName:ea8500af-fe54-4c1e-983d-7a7b82d866b3 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:28.632057903 +0000 UTC m=+6.150567904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret") pod "global-pull-secret-syncer-pbcbk" (UID: "ea8500af-fe54-4c1e-983d-7a7b82d866b3") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:26.632602 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:26.632576 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:26.632602 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:26.632600 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:26.632726 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:26.632615 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q6nsz for pod openshift-network-diagnostics/network-check-target-x4786: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:26.632726 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:26.632661 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz podName:95cc5486-946f-4ada-be5d-84fc0068d031 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:28.632646892 +0000 UTC m=+6.151156869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6nsz" (UniqueName: "kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz") pod "network-check-target-x4786" (UID: "95cc5486-946f-4ada-be5d-84fc0068d031") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:27.005786 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:27.005693 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:34:25 +0000 UTC" deadline="2028-07-11 08:23:10.673715994 +0000 UTC" Nov 02 17:39:27.005786 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:27.005753 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14798h43m43.667967232s" Nov 02 17:39:27.080504 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:27.080448 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:27.080678 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:27.080621 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:27.082116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:27.081165 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:27.082116 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:27.081297 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:27.082116 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:27.081376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:27.082116 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:27.081444 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:27.173033 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:27.172975 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" event={"ID":"06cff5a60959fbf288446179900e7d21","Type":"ContainerStarted","Data":"d847a36e01dcca44e02761f4217fd1f7a2657e5faea23383b1bf45b52a719b26"} Nov 02 17:39:28.180365 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:28.180049 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b8fb63764518311609bba29a4e6d416" containerID="5a2ab06b544a5542188bfd0ad8564f2fec1bf331ceccb794f41ad056b0195861" exitCode=0 Nov 02 17:39:28.180921 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:28.180213 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" event={"ID":"3b8fb63764518311609bba29a4e6d416","Type":"ContainerDied","Data":"5a2ab06b544a5542188bfd0ad8564f2fec1bf331ceccb794f41ad056b0195861"} Nov 02 17:39:28.201524 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:28.200118 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" podStartSLOduration=4.200096424 podStartE2EDuration="4.200096424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:27.193839918 +0000 UTC m=+4.712349922" watchObservedRunningTime="2026-11-02 17:39:28.200096424 +0000 UTC m=+5.718606426" Nov 02 17:39:28.547774 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:28.547511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:28.547774 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:28.547689 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:28.547774 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:28.547750 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs podName:9edee41d-f2cc-404e-ae90-33dd5312d523 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:32.547730749 +0000 UTC m=+10.066240741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs") pod "network-metrics-daemon-wtm26" (UID: "9edee41d-f2cc-404e-ae90-33dd5312d523") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:28.649194 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:28.648397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nsz\" (UniqueName: \"kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz\") pod \"network-check-target-x4786\" (UID: \"95cc5486-946f-4ada-be5d-84fc0068d031\") " pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:28.649194 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:28.648447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:28.649194 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:28.648568 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:28.649194 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:28.648627 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret podName:ea8500af-fe54-4c1e-983d-7a7b82d866b3 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:32.648605544 +0000 UTC m=+10.167115523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret") pod "global-pull-secret-syncer-pbcbk" (UID: "ea8500af-fe54-4c1e-983d-7a7b82d866b3") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:28.649194 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:28.649080 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:28.649194 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:28.649094 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:28.649194 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:28.649103 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q6nsz for pod openshift-network-diagnostics/network-check-target-x4786: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:28.649194 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:28.649136 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz podName:95cc5486-946f-4ada-be5d-84fc0068d031 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:32.64912627 +0000 UTC m=+10.167636247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6nsz" (UniqueName: "kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz") pod "network-check-target-x4786" (UID: "95cc5486-946f-4ada-be5d-84fc0068d031") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:29.082661 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:29.082615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:29.082867 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:29.082765 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:29.083368 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:29.083345 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:29.083558 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:29.083466 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:29.083708 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:29.083603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:29.083708 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:29.083695 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:29.192415 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:29.191536 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="ad0d0893b9bdec64f80270e29e16a6a35d9c346293bf9be884af29574abdfb91" exitCode=0 Nov 02 17:39:29.192415 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:29.191845 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8c9g" event={"ID":"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e","Type":"ContainerDied","Data":"ad0d0893b9bdec64f80270e29e16a6a35d9c346293bf9be884af29574abdfb91"} Nov 02 17:39:29.202937 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:29.202291 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" event={"ID":"3b8fb63764518311609bba29a4e6d416","Type":"ContainerStarted","Data":"ad829fb1d1dd376ccdd57a0e3b7cde1a91ae253c6640fe0969e62b5cecebfc38"} Nov 02 17:39:30.210365 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:30.210258 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-586q5" event={"ID":"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13","Type":"ContainerStarted","Data":"be51c32f51b6cca6b7014d8920877b4bb05b3542ab43263e03ed48c91a84b261"} Nov 02 17:39:30.229655 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:30.228529 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" podStartSLOduration=6.228507093 podStartE2EDuration="6.228507093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:29.243268537 +0000 UTC m=+6.761778543" watchObservedRunningTime="2026-11-02 17:39:30.228507093 +0000 UTC m=+7.747017095" Nov 02 17:39:30.229655 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:30.229003 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-586q5" podStartSLOduration=3.004534103 podStartE2EDuration="7.228994425s" totalImagesPullingTime="4.224460322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:30.22836542 +0000 UTC m=+7.746875496" watchObservedRunningTime="2026-11-02 17:39:30.228994425 +0000 UTC m=+7.747504428" Nov 02 17:39:30.268238 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:30.268204 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:30.268811 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:30.268787 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:31.091870 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:31.091778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:31.092036 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:31.091913 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:31.092353 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:31.092334 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:31.092464 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:31.092444 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:31.092524 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:31.092509 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:31.092648 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:31.092591 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:31.218226 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:31.218187 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2crqk" event={"ID":"a3b7e801-25bc-4f6e-be57-f3131d2ce39c","Type":"ContainerStarted","Data":"29a95e6d9e05f16f563a80db265321a5986cbd800816d9bc7a1b13d56841d80d"} Nov 02 17:39:32.223211 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:32.222719 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" event={"ID":"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e","Type":"ContainerStarted","Data":"c9b681ed630d8dd2da8981768723767cf4f0613bfbb41b787484663c91885d14"} Nov 02 17:39:32.223211 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:32.222816 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:39:32.583071 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:32.582432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:32.583071 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:32.582621 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:32.583071 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:32.582684 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs podName:9edee41d-f2cc-404e-ae90-33dd5312d523 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:40.582666319 +0000 UTC m=+18.101176309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs") pod "network-metrics-daemon-wtm26" (UID: "9edee41d-f2cc-404e-ae90-33dd5312d523") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:32.683914 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:32.683881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nsz\" (UniqueName: \"kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz\") pod \"network-check-target-x4786\" (UID: \"95cc5486-946f-4ada-be5d-84fc0068d031\") " pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:32.684105 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:32.683935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:32.684105 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:32.684048 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:32.684217 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:32.684125 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret podName:ea8500af-fe54-4c1e-983d-7a7b82d866b3 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:40.684108002 +0000 UTC m=+18.202617997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret") pod "global-pull-secret-syncer-pbcbk" (UID: "ea8500af-fe54-4c1e-983d-7a7b82d866b3") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:32.684535 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:32.684515 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:32.684629 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:32.684540 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:32.684629 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:32.684553 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q6nsz for pod openshift-network-diagnostics/network-check-target-x4786: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:32.684629 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:32.684594 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz podName:95cc5486-946f-4ada-be5d-84fc0068d031 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:40.684580813 +0000 UTC m=+18.203090801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6nsz" (UniqueName: "kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz") pod "network-check-target-x4786" (UID: "95cc5486-946f-4ada-be5d-84fc0068d031") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:33.084017 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:33.083976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:33.084198 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:33.084135 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:33.084552 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:33.084529 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:33.084647 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:33.084615 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:33.084729 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:33.084672 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:33.084780 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:33.084743 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:33.521925 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:33.521896 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:39:33.978494 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:33.978367 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:39:33.521923572Z","UUID":"f0f8959a-a7ff-4e7e-8977-eb75bfa8c599","Handler":null,"Name":"","Endpoint":""} Nov 02 17:39:33.980048 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:33.980022 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:39:33.980048 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:33.980047 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:39:34.230198 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:34.229949 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" event={"ID":"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e","Type":"ContainerStarted","Data":"66a3120f2eca98109872c2de3e6c2727a57b6d4bf514f0e7913f9e2dfffd48a3"} Nov 02 17:39:34.231989 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:34.231959 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-59bfz" event={"ID":"4307ecfe-be36-4891-9c9d-4c649e9aca70","Type":"ContainerStarted","Data":"f1cbdfcad00ba045ed91b0bc85994c887ada86abf2c92a3bcf231c92c447c3c3"} Nov 02 17:39:34.234008 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:34.233973 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ksps4" event={"ID":"b63aff75-00c5-4a5c-b9d3-678e6ed4465f","Type":"ContainerStarted","Data":"8a2987240db418328caa727ee215f67cffaba1dda671fe926242758246bcf654"} Nov 02 17:39:34.235964 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:34.235937 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="bf9a1e767eaa102a72c60c391b1b6307fe7cb5a23d7d77b05c1da85d17148698" exitCode=0 Nov 02 17:39:34.236070 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:34.235969 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8c9g" event={"ID":"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e","Type":"ContainerDied","Data":"bf9a1e767eaa102a72c60c391b1b6307fe7cb5a23d7d77b05c1da85d17148698"} Nov 02 17:39:34.254555 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:34.254504 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2crqk" podStartSLOduration=5.6487412070000005 podStartE2EDuration="11.254486142s" totalImagesPullingTime="5.605744935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:31.23837128 +0000 UTC m=+8.756881285" watchObservedRunningTime="2026-11-02 17:39:34.254486142 +0000 UTC m=+11.772996142" Nov 02 17:39:34.276313 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:34.276264 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-59bfz" podStartSLOduration=2.6998314260000003 podStartE2EDuration="11.27624712s" totalImagesPullingTime="8.576415694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:34.255111267 +0000 UTC m=+11.773621265" watchObservedRunningTime="2026-11-02 17:39:34.27624712 +0000 UTC m=+11.794757118" Nov 02 17:39:34.305989 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:34.305936 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ksps4" podStartSLOduration=2.98980988 podStartE2EDuration="11.305916902s" totalImagesPullingTime="8.316107022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:34.27733831 +0000 UTC m=+11.795848307" watchObservedRunningTime="2026-11-02 17:39:34.305916902 +0000 UTC m=+11.824426922" Nov 02 17:39:35.080223 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:35.080185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:35.080223 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:35.080199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:35.080670 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:35.080401 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:35.080670 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:35.080427 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:35.080670 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:35.080543 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:35.080765 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:35.080677 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:35.241887 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:35.241851 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="978781b604e8c2dd6dad49af9e2d9cb86fb42d2322f95bce919261e70e0a445f" exitCode=0 Nov 02 17:39:35.242094 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:35.241930 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8c9g" event={"ID":"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e","Type":"ContainerDied","Data":"978781b604e8c2dd6dad49af9e2d9cb86fb42d2322f95bce919261e70e0a445f"} Nov 02 17:39:35.244985 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:35.244937 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z7lsw" event={"ID":"61d00064-21c5-44b2-a467-18d93e752363","Type":"ContainerStarted","Data":"416627e54a8c22d960f0cb86969d9b2e0e6b9c691c236cd076e3a9aa5964deae"} Nov 02 17:39:35.248479 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:35.248456 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" event={"ID":"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e","Type":"ContainerStarted","Data":"c0f4d73bbfe844c6175be1adf814ba628b7718bde7a1d204a5011fd2310e0931"} Nov 02 17:39:35.288830 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:35.288781 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" podStartSLOduration=3.436436401 podStartE2EDuration="12.288768613s" totalImagesPullingTime="8.852332212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:35.288494741 +0000 UTC m=+12.807004739" watchObservedRunningTime="2026-11-02 17:39:35.288768613 +0000 UTC m=+12.807278612" Nov 02 17:39:35.308334 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:35.307888 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z7lsw" podStartSLOduration=3.933539509 podStartE2EDuration="12.307874996s" totalImagesPullingTime="8.374335487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:35.307381301 +0000 UTC m=+12.825891300" watchObservedRunningTime="2026-11-02 17:39:35.307874996 +0000 UTC m=+12.826384992" Nov 02 17:39:36.252863 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:36.252827 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" event={"ID":"9762673b-765b-4d8c-8791-ee16e740c1b3","Type":"ContainerStarted","Data":"2a8f967545d0f6d034c51fbb7935b0b420a34dffb3fbc79a09fac8aa8551509a"} Nov 02 17:39:36.285301 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:36.285254 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" podStartSLOduration=2.420285222 podStartE2EDuration="13.285240137s" totalImagesPullingTime="10.864954915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:36.280255052 +0000 UTC m=+13.798765048" watchObservedRunningTime="2026-11-02 17:39:36.285240137 +0000 UTC m=+13.803750133" Nov 02 17:39:37.080713 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:37.080679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:37.080713 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:37.080716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:37.080957 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:37.080716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:37.080957 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:37.080818 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:37.081044 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:37.080960 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:37.081462 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:37.081434 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:37.257388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:37.257352 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="06dbb2d07a9415751e290d09e5ce1cb079bfdf4aac18e6b2b0f4a0f849361d61" exitCode=0 Nov 02 17:39:37.257985 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:37.257416 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8c9g" event={"ID":"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e","Type":"ContainerDied","Data":"06dbb2d07a9415751e290d09e5ce1cb079bfdf4aac18e6b2b0f4a0f849361d61"} Nov 02 17:39:39.082958 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:39.082700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:39.083440 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:39.082893 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:39.083440 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:39.083061 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:39.083440 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:39.083132 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:39.083440 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:39.082931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:39.083440 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:39.083222 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:39.265331 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:39.265297 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="6db777e4d072fb23d30426d7e201e6979043b84fd186bf73fc65bac93a8a7903" exitCode=0 Nov 02 17:39:39.265465 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:39.265383 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8c9g" event={"ID":"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e","Type":"ContainerDied","Data":"6db777e4d072fb23d30426d7e201e6979043b84fd186bf73fc65bac93a8a7903"} Nov 02 17:39:40.272834 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:40.272801 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="43ce40ba52fa12694bff473c36eb577cb7bdbcbfbb21a52eda21042fd0b8d21b" exitCode=0 Nov 02 17:39:40.273289 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:40.272849 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8c9g" event={"ID":"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e","Type":"ContainerDied","Data":"43ce40ba52fa12694bff473c36eb577cb7bdbcbfbb21a52eda21042fd0b8d21b"} Nov 02 17:39:40.647586 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:40.647562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:40.647694 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:40.647680 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:40.647750 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:40.647742 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs podName:9edee41d-f2cc-404e-ae90-33dd5312d523 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:56.647722654 +0000 UTC m=+34.166232643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs") pod "network-metrics-daemon-wtm26" (UID: "9edee41d-f2cc-404e-ae90-33dd5312d523") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:39:40.747942 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:40.747901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:40.748138 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:40.747960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nsz\" (UniqueName: \"kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz\") pod \"network-check-target-x4786\" (UID: \"95cc5486-946f-4ada-be5d-84fc0068d031\") " pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:40.748138 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:40.748103 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:39:40.748138 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:40.748104 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:40.748298 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:40.748188 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret podName:ea8500af-fe54-4c1e-983d-7a7b82d866b3 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:56.748167708 +0000 UTC m=+34.266677687 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret") pod "global-pull-secret-syncer-pbcbk" (UID: "ea8500af-fe54-4c1e-983d-7a7b82d866b3") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:39:40.748298 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:40.748120 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:39:40.748298 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:40.748230 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q6nsz for pod openshift-network-diagnostics/network-check-target-x4786: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:40.748298 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:40.748279 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz podName:95cc5486-946f-4ada-be5d-84fc0068d031 nodeName:}" failed. No retries permitted until 2026-11-02 17:39:56.748266601 +0000 UTC m=+34.266776588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6nsz" (UniqueName: "kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz") pod "network-check-target-x4786" (UID: "95cc5486-946f-4ada-be5d-84fc0068d031") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:39:41.080408 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.080376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:41.080408 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.080399 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:41.080635 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.080428 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:41.080635 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:41.080496 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:41.080635 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:41.080573 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:41.080728 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:41.080656 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:41.265291 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.265251 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:41.265456 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.265392 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:39:41.265904 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.265883 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-586q5" Nov 02 17:39:41.276917 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.276891 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" event={"ID":"d417b861-3a8a-4889-8a97-3a03132e8623","Type":"ContainerStarted","Data":"ee3e505c24c136d00f2efed3a7972b060cc6bb931eb4569624cc27230d20943e"} Nov 02 17:39:41.277298 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.276952 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" event={"ID":"d417b861-3a8a-4889-8a97-3a03132e8623","Type":"ContainerStarted","Data":"6955dce1332f7b16736c8038996cea3635f9e110801b110719612a34bda6b6a9"} Nov 02 17:39:41.277298 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.276997 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" event={"ID":"d417b861-3a8a-4889-8a97-3a03132e8623","Type":"ContainerStarted","Data":"a1a7fc2664687f05b72b116e1f4c040dd9bca2c84aa09bd89fe502486e0eae05"} Nov 02 17:39:41.277298 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.277031 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" event={"ID":"d417b861-3a8a-4889-8a97-3a03132e8623","Type":"ContainerStarted","Data":"f237488bcb9f0e4bd1d778d288c9608b53d8266b292043ed426bc9a71b4124c8"} Nov 02 17:39:41.277298 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.277062 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" event={"ID":"d417b861-3a8a-4889-8a97-3a03132e8623","Type":"ContainerStarted","Data":"cab5b870d800572bec53113387b8873fc8c1751f21ab7190829664f24e72fbf5"} Nov 02 17:39:41.277298 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.277109 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" event={"ID":"d417b861-3a8a-4889-8a97-3a03132e8623","Type":"ContainerStarted","Data":"e8a01b47fed516e8e07b694162f45038a82b5742eb4053c2d501250b91c89af6"} Nov 02 17:39:41.280222 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.280200 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l8c9g" event={"ID":"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e","Type":"ContainerStarted","Data":"62ea1480bc0f1d635273f04faec99a1cb99d26a61ab32b18fc0bd32dda212b19"} Nov 02 17:39:41.310789 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:41.310735 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l8c9g" podStartSLOduration=7.053178468 podStartE2EDuration="18.310720813s" totalImagesPullingTime="11.257542345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:41.310102496 +0000 UTC m=+18.828612490" watchObservedRunningTime="2026-11-02 17:39:41.310720813 +0000 UTC m=+18.829230809" Nov 02 17:39:43.083168 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:43.083072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:43.083674 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:43.083169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:43.083674 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:43.083258 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:43.083674 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:43.083252 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:43.083674 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:43.083330 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:43.083674 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:43.083393 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:43.287124 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:43.287090 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" event={"ID":"d417b861-3a8a-4889-8a97-3a03132e8623","Type":"ContainerStarted","Data":"12eee4ae87be8fc9335a1865ec5738a8edcffd0e2e09f89c711295a601fdadba"} Nov 02 17:39:44.447635 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:44.447595 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:44.617539 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:44.617500 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:45.083126 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:45.083063 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:45.083126 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:45.083097 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:45.083126 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:45.083115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:45.083354 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:45.083200 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:45.083354 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:45.083309 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:45.083410 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:45.083374 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:45.295220 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:45.295183 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" event={"ID":"d417b861-3a8a-4889-8a97-3a03132e8623","Type":"ContainerStarted","Data":"94b9b11a382038ad900432a9f960fe88ab82823c0eef531e1edb59f564d029e2"} Nov 02 17:39:45.295584 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:45.295560 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:45.311171 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:45.311147 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:45.325849 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:45.325810 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" podStartSLOduration=7.192837775 podStartE2EDuration="22.325798195s" totalImagesPullingTime="15.13296042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:45.325720514 +0000 UTC m=+22.844230510" watchObservedRunningTime="2026-11-02 17:39:45.325798195 +0000 UTC m=+22.844308192" Nov 02 17:39:46.297209 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:46.297184 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:39:46.297790 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:46.297611 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:46.312841 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:46.312819 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:46.717643 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:46.717616 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pbcbk"] Nov 02 17:39:46.717865 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:46.717712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:46.717865 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:46.717811 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:46.722677 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:46.722647 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wtm26"] Nov 02 17:39:46.722833 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:46.722784 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:46.723009 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:46.722917 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:46.723230 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:46.723208 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x4786"] Nov 02 17:39:46.723323 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:46.723297 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:46.723819 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:46.723355 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:47.299004 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:47.298962 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:39:48.080676 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:48.080644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:48.080676 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:48.080675 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:48.080880 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:48.080748 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:48.080880 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:48.080823 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:48.300661 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:48.300638 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:39:48.461258 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:48.461159 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:48.565874 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:48.565842 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:49.080900 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:49.080863 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:49.081064 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:49.080976 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:50.080843 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:50.080805 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:50.081317 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:50.080805 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:50.081317 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:50.080928 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:50.081317 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:50.081019 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:51.080556 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:51.080522 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:51.080808 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:51.080630 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:51.236898 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:51.236872 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:39:51.237322 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:51.237052 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:39:51.243561 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:51.243535 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:51.247632 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:51.247601 2580 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" podUID="d417b861-3a8a-4889-8a97-3a03132e8623" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 17:39:51.256042 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:51.256016 2580 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" podUID="d417b861-3a8a-4889-8a97-3a03132e8623" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 17:39:51.392246 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:51.392170 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:51.525864 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:51.525832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:51.666022 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:51.665950 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:51.999302 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:51.999266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:52.080574 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:52.080541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:52.080574 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:52.080564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:52.080768 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:52.080636 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pbcbk" podUID="ea8500af-fe54-4c1e-983d-7a7b82d866b3" Nov 02 17:39:52.080768 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:52.080761 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtm26" podUID="9edee41d-f2cc-404e-ae90-33dd5312d523" Nov 02 17:39:52.179471 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:52.179430 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:52.882178 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:52.882148 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:53.081030 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:53.080995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:53.081196 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:53.081089 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4786" podUID="95cc5486-946f-4ada-be5d-84fc0068d031" Nov 02 17:39:53.230534 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:53.230502 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:53.346989 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:53.346964 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeReady" Nov 02 17:39:53.347170 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:53.347100 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 17:39:53.426646 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:53.426608 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:53.564663 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:53.564593 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:53.735487 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:53.735457 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:54.080253 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:54.080214 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:54.080627 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:54.080217 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:54.085620 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:54.085597 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mkftb\"" Nov 02 17:39:54.085620 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:54.085597 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:39:54.086752 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:54.086734 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:39:54.086814 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:54.086739 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7mg2x\"" Nov 02 17:39:54.163772 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:54.163744 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-59bfz_4307ecfe-be36-4891-9c9d-4c649e9aca70/dns-node-resolver/0.log" Nov 02 17:39:54.602957 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:54.602911 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:54.879417 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:54.879336 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:54.981845 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:54.981816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:55.080238 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:55.080201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:55.086991 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:55.086972 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:39:55.087316 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:55.087013 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6x7d8\"" Nov 02 17:39:55.087991 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:55.087973 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:39:55.211630 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:55.211527 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:55.361900 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:55.361875 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2crqk_a3b7e801-25bc-4f6e-be57-f3131d2ce39c/node-ca/0.log" Nov 02 17:39:55.392491 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:55.392464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:55.896945 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:55.896916 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:55.996317 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:55.996284 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:56.196390 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:56.196312 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:56.593346 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:56.593313 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:56.662017 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:56.661975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:39:56.662195 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:56.662142 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 17:39:56.662246 ip-10-0-135-9 kubenswrapper[2580]: E1102 17:39:56.662207 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs podName:9edee41d-f2cc-404e-ae90-33dd5312d523 nodeName:}" failed. No retries permitted until 2026-11-02 17:40:28.662192072 +0000 UTC m=+66.180702048 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs") pod "network-metrics-daemon-wtm26" (UID: "9edee41d-f2cc-404e-ae90-33dd5312d523") : secret "metrics-daemon-secret" not found Nov 02 17:39:56.763239 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:56.763205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nsz\" (UniqueName: \"kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz\") pod \"network-check-target-x4786\" (UID: \"95cc5486-946f-4ada-be5d-84fc0068d031\") " pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:56.763400 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:56.763248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:56.767213 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:56.767184 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:56.767378 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:56.767360 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6nsz\" (UniqueName: \"kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz\") pod \"network-check-target-x4786\" (UID: \"95cc5486-946f-4ada-be5d-84fc0068d031\") " pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:56.809555 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:56.809521 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:56.856622 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:56.856594 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:39:56.917597 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:56.917572 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:39:57.200367 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:57.200302 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:57.215559 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:57.215527 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x4786"] Nov 02 17:39:57.218977 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:57.218950 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cc5486_946f_4ada_be5d_84fc0068d031.slice/crio-b54e22c2a99faf8975976fa32ca0768be32ef355499d057800cb384fdb520f86 WatchSource:0}: Error finding container b54e22c2a99faf8975976fa32ca0768be32ef355499d057800cb384fdb520f86: Status 404 returned error can't find the container with id b54e22c2a99faf8975976fa32ca0768be32ef355499d057800cb384fdb520f86 Nov 02 17:39:57.220308 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:57.220288 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pbcbk"] Nov 02 17:39:57.223296 ip-10-0-135-9 kubenswrapper[2580]: W1102 17:39:57.223273 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8500af_fe54_4c1e_983d_7a7b82d866b3.slice/crio-688f509d6c74b21ba8392f7aea20f7e5768fdf28f9f5f8ac380840e4f0716dbc WatchSource:0}: Error finding container 688f509d6c74b21ba8392f7aea20f7e5768fdf28f9f5f8ac380840e4f0716dbc: Status 404 returned error can't find the container with id 688f509d6c74b21ba8392f7aea20f7e5768fdf28f9f5f8ac380840e4f0716dbc Nov 02 17:39:57.302996 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:57.302968 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:57.320670 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:57.320639 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x4786" event={"ID":"95cc5486-946f-4ada-be5d-84fc0068d031","Type":"ContainerStarted","Data":"b54e22c2a99faf8975976fa32ca0768be32ef355499d057800cb384fdb520f86"} Nov 02 17:39:57.321769 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:57.321750 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pbcbk" event={"ID":"ea8500af-fe54-4c1e-983d-7a7b82d866b3","Type":"ContainerStarted","Data":"688f509d6c74b21ba8392f7aea20f7e5768fdf28f9f5f8ac380840e4f0716dbc"} Nov 02 17:39:57.403853 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:57.403815 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:57.629240 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:57.629206 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:57.720215 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:57.720173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:58.003531 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:58.003494 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:58.609839 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:58.609800 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:58.724373 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:58.724335 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:58.813707 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:58.813673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:58.914969 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:58.914884 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:59.015289 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:59.015248 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:59.115102 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:59.115053 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:59.215641 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:59.215613 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:59.383688 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:59.383606 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:59.573472 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:59.573377 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:39:59.986662 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:39:59.986625 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:00.588878 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:00.588842 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:00.782268 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:00.782234 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:00.959477 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:00.959382 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:01.075819 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:01.075778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:01.272340 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:01.272311 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:01.334649 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:01.334612 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x4786" event={"ID":"95cc5486-946f-4ada-be5d-84fc0068d031","Type":"ContainerStarted","Data":"58ae7cf7c378d05bc1f097270237c893b6cf2d7beda7b201f85f7442ecac9a16"} Nov 02 17:40:01.334952 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:01.334849 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:40:01.353632 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:01.353578 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x4786" podStartSLOduration=34.901252717 podStartE2EDuration="38.353561921s" totalImagesPullingTime="3.452309204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:01.352893797 +0000 UTC m=+38.871403806" watchObservedRunningTime="2026-11-02 17:40:01.353561921 +0000 UTC m=+38.872071919" Nov 02 17:40:01.413136 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:01.413108 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:01.512895 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:01.512845 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:01.619892 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:01.619812 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:01.789752 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:01.789720 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:01.891835 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:01.891762 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:01.994729 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:01.994699 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:02.338388 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:02.338347 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pbcbk" event={"ID":"ea8500af-fe54-4c1e-983d-7a7b82d866b3","Type":"ContainerStarted","Data":"05b529bf96f62fef937220780e237fda0d1a7a066ce72ff2300d685010d66fb3"} Nov 02 17:40:02.358641 ip-10-0-135-9 kubenswrapper[2580]: I1102 17:40:02.358578 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pbcbk" podStartSLOduration=34.379096758 podStartE2EDuration="39.358561223s" totalImagesPullingTime="4.979464465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:02.357091467 +0000 UTC m=+39.875601456" watchObservedRunningTime="2026-11-02 17:40:02.358561223 +0000 UTC m=+39.877071220" Nov 02 17:40:02.492377 ip-10-0-135-9 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 17:40:02.515510 ip-10-0-135-9 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 17:40:02.515684 ip-10-0-135-9 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 17:40:02.515733 ip-10-0-135-9 systemd[1]: kubelet.service: Consumed 2.208s CPU time. Nov 02 17:40:02.526756 ip-10-0-135-9 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:40:02.595232 ip-10-0-135-9 kubenswrapper[4310]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:40:02.595232 ip-10-0-135-9 kubenswrapper[4310]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:40:02.595232 ip-10-0-135-9 kubenswrapper[4310]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:40:02.595232 ip-10-0-135-9 kubenswrapper[4310]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:40:02.597461 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597370 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:02.597461 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597456 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:02.597461 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597462 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:02.597461 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597465 4310 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:02.597461 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597469 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:02.597461 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597472 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597475 4310 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597478 4310 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597480 4310 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597484 4310 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597487 4310 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597490 4310 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597492 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597495 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597498 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597501 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597504 4310 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597506 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597509 4310 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597518 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597522 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597524 4310 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597527 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597530 4310 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:02.597717 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597532 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597535 4310 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597537 4310 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597540 4310 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597543 4310 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597546 4310 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597549 4310 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597552 4310 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597554 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597557 4310 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597560 4310 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597562 4310 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597565 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597568 4310 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597571 4310 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597574 4310 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597577 4310 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597579 4310 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597582 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597585 4310 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:02.598235 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597588 4310 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597591 4310 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597594 4310 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597596 4310 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597599 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597602 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597604 4310 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597607 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597609 4310 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597612 4310 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597615 4310 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597617 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597620 4310 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597622 4310 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597625 4310 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597627 4310 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597630 4310 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597632 4310 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597635 4310 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597638 4310 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:02.598815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597641 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597646 4310 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597650 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597653 4310 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597656 4310 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597659 4310 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597662 4310 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597664 4310 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597669 4310 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597673 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597676 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597679 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597681 4310 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597684 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597687 4310 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597689 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597692 4310 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597695 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597698 4310 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:02.599395 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597701 4310 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597703 4310 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597706 4310 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597708 4310 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597711 4310 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597714 4310 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597716 4310 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597720 4310 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597722 4310 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597725 4310 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597728 4310 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597734 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597738 4310 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597741 4310 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597743 4310 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597746 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597749 4310 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597752 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597754 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597757 4310 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:02.599889 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597759 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597762 4310 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.597765 4310 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598580 4310 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598587 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598590 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598593 4310 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598596 4310 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598608 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598611 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598614 4310 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598617 4310 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598620 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598623 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598626 4310 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598629 4310 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598632 4310 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598634 4310 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598637 4310 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598640 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:02.600420 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598642 4310 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598645 4310 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598647 4310 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598653 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598656 4310 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598659 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598662 4310 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598665 4310 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598670 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598676 4310 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598680 4310 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598684 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598688 4310 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598692 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598697 4310 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598700 4310 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598703 4310 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598705 4310 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598708 4310 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598711 4310 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:02.600938 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598713 4310 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598716 4310 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598719 4310 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598722 4310 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598724 4310 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598727 4310 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598730 4310 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598733 4310 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598735 4310 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598738 4310 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598740 4310 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598745 4310 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598749 4310 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598751 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598754 4310 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598759 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598761 4310 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598764 4310 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598767 4310 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:02.601512 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598769 4310 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598774 4310 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598779 4310 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598782 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598785 4310 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598788 4310 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598790 4310 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598793 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598795 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598798 4310 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598801 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598803 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598806 4310 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598809 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598811 4310 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598814 4310 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598816 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598819 4310 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598822 4310 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:02.601987 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598825 4310 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598827 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598831 4310 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598833 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598836 4310 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598838 4310 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598841 4310 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598844 4310 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598846 4310 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598850 4310 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598853 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598855 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598858 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598861 4310 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598863 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598865 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598869 4310 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598872 4310 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598874 4310 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:02.602529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598877 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598879 4310 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598882 4310 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598884 4310 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598887 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598889 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598892 4310 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598894 4310 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598897 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598901 4310 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598904 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.598906 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599458 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599465 4310 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599468 4310 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599471 4310 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599474 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599477 4310 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599480 4310 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599482 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:02.603036 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599485 4310 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599488 4310 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599492 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599495 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599498 4310 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599501 4310 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599503 4310 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599506 4310 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599509 4310 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599512 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599514 4310 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599517 4310 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599520 4310 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599522 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599525 4310 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599528 4310 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599530 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599533 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599535 4310 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599538 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:02.603844 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599541 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599543 4310 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599546 4310 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599548 4310 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599551 4310 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599553 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599556 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599559 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599561 4310 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599564 4310 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599567 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599569 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599572 4310 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599574 4310 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599578 4310 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599581 4310 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599583 4310 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599586 4310 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599588 4310 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599591 4310 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:02.604790 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599593 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599596 4310 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599599 4310 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599602 4310 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599604 4310 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599607 4310 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599611 4310 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599615 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599619 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599622 4310 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599625 4310 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599627 4310 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599630 4310 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599633 4310 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599636 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599639 4310 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599641 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599644 4310 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599646 4310 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:02.605951 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599649 4310 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599653 4310 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599657 4310 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599660 4310 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599663 4310 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599665 4310 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599668 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599672 4310 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599675 4310 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599677 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599680 4310 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599682 4310 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599685 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599687 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599690 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599692 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599696 4310 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599698 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599701 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:02.606465 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599704 4310 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599706 4310 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599709 4310 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599711 4310 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599714 4310 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599716 4310 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599719 4310 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599722 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599724 4310 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599726 4310 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599729 4310 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599731 4310 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599734 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599736 4310 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599739 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599742 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599745 4310 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599747 4310 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599750 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.599754 4310 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599848 4310 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:40:02.606966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599855 4310 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599862 4310 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599868 4310 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599872 4310 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599876 4310 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599881 4310 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599885 4310 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599889 4310 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599893 4310 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599897 4310 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599901 4310 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599904 4310 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599908 4310 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599911 4310 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599914 4310 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599917 4310 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599920 4310 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599925 4310 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599928 4310 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599931 4310 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599934 4310 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599938 4310 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599942 4310 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:40:02.607503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599945 4310 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599948 4310 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599952 4310 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599955 4310 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599958 4310 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599961 4310 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599964 4310 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599969 4310 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599974 4310 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599977 4310 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599980 4310 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599983 4310 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599986 4310 flags.go:64] FLAG: --enable-server="true" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599989 4310 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599995 4310 flags.go:64] FLAG: --event-burst="100" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.599998 4310 flags.go:64] FLAG: --event-qps="50" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600001 4310 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600004 4310 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600007 4310 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600011 4310 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600015 4310 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600018 4310 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600021 4310 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600024 4310 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600027 4310 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:40:02.608053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600030 4310 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600033 4310 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600036 4310 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600039 4310 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600042 4310 flags.go:64] FLAG: --feature-gates="" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600046 4310 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600050 4310 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600053 4310 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600056 4310 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600059 4310 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600062 4310 flags.go:64] FLAG: --help="false" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600065 4310 flags.go:64] FLAG: --hostname-override="ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600069 4310 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600072 4310 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600089 4310 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600093 4310 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600097 4310 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600100 4310 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600103 4310 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600106 4310 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600109 4310 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600112 4310 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600116 4310 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600118 4310 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:40:02.608692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600121 4310 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600124 4310 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600127 4310 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600131 4310 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600134 4310 flags.go:64] FLAG: --lock-file="" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600137 4310 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600140 4310 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600143 4310 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600149 4310 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600152 4310 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600155 4310 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600158 4310 flags.go:64] FLAG: --logging-format="text" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600161 4310 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600165 4310 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600168 4310 flags.go:64] FLAG: --manifest-url="" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600171 4310 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600176 4310 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600179 4310 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600183 4310 flags.go:64] FLAG: --max-pods="110" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600186 4310 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600189 4310 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600192 4310 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600197 4310 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600200 4310 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600203 4310 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:40:02.609419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600206 4310 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600214 4310 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600217 4310 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600220 4310 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600223 4310 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600226 4310 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600229 4310 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600232 4310 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600235 4310 flags.go:64] FLAG: --port="10250" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600238 4310 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600242 4310 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ab4fbfc104dbe07a" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600245 4310 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600248 4310 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600252 4310 flags.go:64] FLAG: --register-node="true" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600254 4310 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600258 4310 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600261 4310 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600264 4310 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600272 4310 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600276 4310 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600279 4310 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600282 4310 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600285 4310 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600288 4310 flags.go:64] FLAG: --runonce="false" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600291 4310 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600294 4310 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600298 4310 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:40:02.610114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600300 4310 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600304 4310 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600308 4310 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600311 4310 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600314 4310 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600317 4310 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600320 4310 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600323 4310 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600327 4310 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600330 4310 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600333 4310 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600336 4310 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600341 4310 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600344 4310 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600347 4310 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600351 4310 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600354 4310 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600357 4310 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600360 4310 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600363 4310 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600366 4310 flags.go:64] FLAG: --v="2" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600371 4310 flags.go:64] FLAG: --version="false" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600375 4310 flags.go:64] FLAG: --vmodule="" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600379 4310 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600384 4310 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:40:02.610795 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600519 4310 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600523 4310 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600526 4310 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600528 4310 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600531 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600534 4310 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600536 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600539 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600541 4310 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600545 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600548 4310 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600551 4310 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600554 4310 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600557 4310 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600559 4310 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600562 4310 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600564 4310 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600567 4310 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600570 4310 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:02.611631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600572 4310 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600575 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600578 4310 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600580 4310 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600583 4310 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600586 4310 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600589 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600592 4310 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600594 4310 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600597 4310 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600599 4310 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600602 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600608 4310 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600611 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600613 4310 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600616 4310 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600619 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600621 4310 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600624 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600627 4310 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:02.612241 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600630 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600633 4310 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600637 4310 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600639 4310 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600642 4310 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600645 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600648 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600650 4310 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600653 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600656 4310 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600658 4310 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600661 4310 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600663 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600666 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600669 4310 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600671 4310 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600674 4310 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600677 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600680 4310 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600683 4310 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:02.612806 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600685 4310 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600688 4310 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600691 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600693 4310 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600698 4310 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600700 4310 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600703 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600706 4310 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600708 4310 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600711 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600713 4310 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600716 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600719 4310 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600721 4310 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600725 4310 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600727 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600730 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600733 4310 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600736 4310 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:02.613387 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600739 4310 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600744 4310 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600748 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600752 4310 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600755 4310 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600758 4310 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600761 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600764 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600766 4310 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600769 4310 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600772 4310 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600775 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600777 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600780 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600783 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600785 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600788 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600792 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600794 4310 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:02.613994 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600797 4310 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600800 4310 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600802 4310 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600805 4310 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600807 4310 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600810 4310 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600812 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600815 4310 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600819 4310 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.600844 4310 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600881 4310 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600886 4310 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600889 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600891 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600894 4310 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600897 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:02.614529 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600899 4310 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600902 4310 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600904 4310 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600907 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600909 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600912 4310 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600915 4310 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600918 4310 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600920 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600923 4310 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600926 4310 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600929 4310 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600931 4310 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600934 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600936 4310 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600940 4310 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600943 4310 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600946 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600949 4310 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600951 4310 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:02.615136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600954 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600957 4310 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600959 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600962 4310 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600966 4310 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600968 4310 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600971 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600974 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600977 4310 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600979 4310 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600983 4310 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600988 4310 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600991 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600994 4310 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.600997 4310 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601000 4310 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601003 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601005 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:02.615654 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601008 4310 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601013 4310 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601016 4310 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601019 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601021 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601024 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601027 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601030 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601033 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601037 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601040 4310 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601042 4310 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601045 4310 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601047 4310 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601050 4310 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601052 4310 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601055 4310 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601057 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601061 4310 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:02.616109 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601064 4310 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601067 4310 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601070 4310 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601072 4310 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601090 4310 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601094 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601099 4310 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601103 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601107 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601111 4310 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601114 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601117 4310 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601119 4310 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601122 4310 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601125 4310 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601127 4310 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601130 4310 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601133 4310 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601135 4310 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601137 4310 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:02.616616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601141 4310 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601143 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601148 4310 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601151 4310 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601153 4310 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601156 4310 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601159 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601161 4310 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601164 4310 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601166 4310 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601169 4310 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601173 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601175 4310 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601178 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601180 4310 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601183 4310 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601185 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601188 4310 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601191 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601193 4310 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:02.617106 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601196 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601199 4310 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.601202 4310 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.606888 4310 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.606905 4310 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.606928 4310 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.606941 4310 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607005 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607011 4310 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607014 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607017 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607020 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607023 4310 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607026 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607028 4310 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607031 4310 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607034 4310 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607037 4310 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607040 4310 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:02.617611 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607043 4310 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607045 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607048 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607050 4310 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607053 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607057 4310 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607061 4310 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607064 4310 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607067 4310 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607070 4310 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607072 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607090 4310 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607095 4310 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607100 4310 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607105 4310 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607118 4310 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607121 4310 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607124 4310 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607126 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:02.618072 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607129 4310 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607132 4310 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607134 4310 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607137 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607140 4310 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607142 4310 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607145 4310 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607148 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607150 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607153 4310 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607155 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607158 4310 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607160 4310 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607163 4310 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607165 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607168 4310 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607171 4310 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607174 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607177 4310 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607179 4310 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:02.618546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607182 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607185 4310 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607187 4310 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607190 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607192 4310 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607196 4310 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607200 4310 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607203 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607206 4310 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607210 4310 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607213 4310 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607215 4310 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607218 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607221 4310 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607223 4310 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607226 4310 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607228 4310 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607231 4310 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607233 4310 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607236 4310 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:02.619037 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607238 4310 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607241 4310 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607244 4310 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607246 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607249 4310 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607251 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607254 4310 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607257 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607261 4310 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607263 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607266 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607268 4310 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607271 4310 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607273 4310 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607276 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607278 4310 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607281 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607284 4310 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607286 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:02.619525 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607289 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607292 4310 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607294 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607297 4310 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607300 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607302 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607305 4310 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607308 4310 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607310 4310 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607313 4310 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607315 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607318 4310 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607320 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607323 4310 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607326 4310 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607328 4310 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:02.620007 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.607355 4310 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607504 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607510 4310 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607513 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607516 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607519 4310 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607523 4310 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607526 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607529 4310 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607531 4310 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607534 4310 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607537 4310 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607540 4310 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607542 4310 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607545 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607547 4310 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607550 4310 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607552 4310 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607555 4310 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607558 4310 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:02.620649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607561 4310 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607564 4310 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607566 4310 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607569 4310 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607571 4310 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607574 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607576 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607579 4310 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607581 4310 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607584 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607586 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607589 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607592 4310 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607594 4310 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607597 4310 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607600 4310 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607602 4310 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607605 4310 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607608 4310 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607611 4310 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:02.621411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607614 4310 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607616 4310 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607618 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607621 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607623 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607626 4310 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607628 4310 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607631 4310 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607634 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607636 4310 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607639 4310 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607641 4310 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607644 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607646 4310 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607649 4310 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607651 4310 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607654 4310 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607656 4310 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607659 4310 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:02.622485 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607661 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607664 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607666 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607669 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607671 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607674 4310 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607677 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607679 4310 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607682 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607685 4310 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607687 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607690 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607692 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607696 4310 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607698 4310 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607701 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607703 4310 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607706 4310 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607709 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:02.623391 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607711 4310 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607714 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607716 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607719 4310 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607721 4310 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607724 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607727 4310 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607730 4310 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607732 4310 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607735 4310 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607737 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607740 4310 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607744 4310 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607747 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607750 4310 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607754 4310 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607758 4310 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607761 4310 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607768 4310 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:02.624292 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607771 4310 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607773 4310 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607779 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607782 4310 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607785 4310 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607788 4310 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607790 4310 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607793 4310 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607796 4310 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607799 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.607822 4310 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607855 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607860 4310 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607863 4310 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607866 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607869 4310 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:40:02.625067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607873 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607876 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607879 4310 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607881 4310 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607884 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607887 4310 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607890 4310 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607892 4310 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607895 4310 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607897 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607900 4310 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607904 4310 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607908 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607911 4310 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607915 4310 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607919 4310 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607924 4310 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607927 4310 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607930 4310 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:40:02.625506 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607933 4310 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607936 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607939 4310 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607942 4310 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607944 4310 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607947 4310 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607950 4310 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607953 4310 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607956 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607958 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607961 4310 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607963 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607966 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607973 4310 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607975 4310 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607978 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607980 4310 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607983 4310 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607986 4310 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607989 4310 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:40:02.625997 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607991 4310 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607994 4310 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607996 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.607999 4310 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608002 4310 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608004 4310 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608007 4310 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608010 4310 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608013 4310 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608017 4310 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608020 4310 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608022 4310 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608025 4310 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608028 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608030 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608033 4310 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608035 4310 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608038 4310 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608040 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608043 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:40:02.626573 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608046 4310 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608048 4310 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608051 4310 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608053 4310 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608056 4310 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608060 4310 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608063 4310 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608066 4310 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608068 4310 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608071 4310 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608086 4310 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608089 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608092 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608094 4310 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608097 4310 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608100 4310 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608102 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608105 4310 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608107 4310 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:40:02.627199 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608110 4310 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608113 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608117 4310 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608119 4310 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608122 4310 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608125 4310 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608127 4310 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608130 4310 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608132 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608135 4310 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608138 4310 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608140 4310 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608143 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608146 4310 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608149 4310 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608152 4310 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608154 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608157 4310 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608160 4310 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:40:02.627690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608162 4310 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608165 4310 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608168 4310 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:02.608171 4310 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.608225 4310 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.609400 4310 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.609464 4310 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.609668 4310 server.go:1014] "Starting client certificate rotation" Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.609810 4310 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.609867 4310 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:34:23 +0000 UTC" deadline="2028-06-11 02:47:11.554589003 +0000 UTC" Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.609895 4310 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14073h7m8.94469567s" Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.610137 4310 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.611196 4310 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.612890 4310 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.614472 4310 log.go:25] "Validated CRI v1 image API" Nov 02 17:40:02.628324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.615243 4310 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:40:02.628690 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.616744 4310 fs.go:135] Filesystem UUIDs: map[309965dd-6c80-4c76-9984-290f457a8012:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c7000704-6e98-4d53-b706-d32bccc558c0:/dev/nvme0n1p3] Nov 02 17:40:02.628733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.616763 4310 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/128194c44a764a57564fb9be1145f552a6eb7f8d040141336abd66f6679d0a01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/128194c44a764a57564fb9be1145f552a6eb7f8d040141336abd66f6679d0a01/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d4854f8417a737d3de81577301ce23da7e44e0f890424ff66a81342462cfb98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d4854f8417a737d3de81577301ce23da7e44e0f890424ff66a81342462cfb98/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53afcb4f0b6a07dad464bc1959e6a377e02b71721178be3604d11ec340930bd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53afcb4f0b6a07dad464bc1959e6a377e02b71721178be3604d11ec340930bd4/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65ff996d599a7608f7fe9d97fa4e09ee14011e646a4240e9c06654e0aa6f23db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65ff996d599a7608f7fe9d97fa4e09ee14011e646a4240e9c06654e0aa6f23db/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/688f509d6c74b21ba8392f7aea20f7e5768fdf28f9f5f8ac380840e4f0716dbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/688f509d6c74b21ba8392f7aea20f7e5768fdf28f9f5f8ac380840e4f0716dbc/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/728f0b46110f27713f0ca1d3a806dc39fa67ac7702e9ee38f172c20af05a5264/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/728f0b46110f27713f0ca1d3a806dc39fa67ac7702e9ee38f172c20af05a5264/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84ead3af633f8a0a68a58e26934899eb447ced88b7d61dc43a6a74e4d844f935/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84ead3af633f8a0a68a58e26934899eb447ced88b7d61dc43a6a74e4d844f935/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4b3bee0a5dafecb36831bd35a2f800a2cf529c89088c55c27e447ed184570b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4b3bee0a5dafecb36831bd35a2f800a2cf529c89088c55c27e447ed184570b3/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b54e22c2a99faf8975976fa32ca0768be32ef355499d057800cb384fdb520f86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b54e22c2a99faf8975976fa32ca0768be32ef355499d057800cb384fdb520f86/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9586f27793af460b023b7e5c1741f9b7cd0d229c3cd46cc5e42631b3bab2bec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9586f27793af460b023b7e5c1741f9b7cd0d229c3cd46cc5e42631b3bab2bec/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3d7515ca0e1a3c98d3379022398065f010623807d452a28d1550585dedfe092/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3d7515ca0e1a3c98d3379022398065f010623807d452a28d1550585dedfe092/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb1271a6e0a5408d6d886fbeb865b4b9275b44bdb4ed32570506aa64b1920993/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb1271a6e0a5408d6d886fbeb865b4b9275b44bdb4ed32570506aa64b1920993/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e/volumes/kubernetes.io~projected/kube-api-access-fbfg9:{mountpoint:/var/lib/kubelet/pods/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e/volumes/kubernetes.io~projected/kube-api-access-fbfg9 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4307ecfe-be36-4891-9c9d-4c649e9aca70/volumes/kubernetes.io~projected/kube-api-access-8rtjz:{mountpoint:/var/lib/kubelet/pods/4307ecfe-be36-4891-9c9d-4c649e9aca70/volumes/kubernetes.io~projected/kube-api-access-8rtjz major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/volumes/kubernetes.io~secret/agent-certs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61d00064-21c5-44b2-a467-18d93e752363/volumes/kubernetes.io~projected/kube-api-access-9swdz:{mountpoint:/var/lib/kubelet/pods/61d00064-21c5-44b2-a467-18d93e752363/volumes/kubernetes.io~projected/kube-api-access-9swdz major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95cc5486-946f-4ada-be5d-84fc0068d031/volumes/kubernetes.io~projected/kube-api-access-q6nsz:{mountpoint:/var/lib/kubelet/pods/95cc5486-946f-4ada-be5d-84fc0068d031/volumes/kubernetes.io~projected/kube-api-access-q6nsz major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9762673b-765b-4d8c-8791-ee16e740c1b3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9762673b-765b-4d8c-8791-ee16e740c1b3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9762673b-765b-4d8c-8791-ee16e740c1b3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9762673b-765b-4d8c-8791-ee16e740c1b3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9762673b-765b-4d8c-8791-ee16e740c1b3/volumes/kubernetes.io~projected/kube-api-access-znq7v:{mountpoint:/var/lib/kubelet/pods/9762673b-765b-4d8c-8791-ee16e740c1b3/volumes/kubernetes.io~projected/kube-api-access-znq7v major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e/volumes/kubernetes.io~projected/kube-api-access-kln72:{mountpoint:/var/lib/kubelet/pods/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e/volumes/kubernetes.io~projected/kube-api-access-kln72 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9edee41d-f2cc-404e-ae90-33dd5312d523/volumes/kubernetes.io~projected/kube-api-access-r8pkt:{mountpoint:/var/lib/kubelet/pods/9edee41d-f2cc-404e-ae90-33dd5312d523/volumes/kubernetes.io~projected/kube-api-access-r8pkt major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3b7e801-25bc-4f6e-be57-f3131d2ce39c/volumes/kubernetes.io~projected/kube-api-access-bwgpc:{mountpoint:/var/lib/kubelet/pods/a3b7e801-25bc-4f6e-be57-f3131d2ce39c/volumes/kubernetes.io~projected/kube-api-access-bwgpc major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b63aff75-00c5-4a5c-b9d3-678e6ed4465f/volumes/kubernetes.io~projected/kube-api-access-qsrdf:{mountpoint:/var/lib/kubelet/pods/b63aff75-00c5-4a5c-b9d3-678e6ed4465f/volumes/kubernetes.io~projected/kube-api-access-qsrdf major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d417b861-3a8a-4889-8a97-3a03132e8623/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d417b861-3a8a-4889-8a97-3a03132e8623/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d417b861-3a8a-4889-8a97-3a03132e8623/volumes/kubernetes.io~projected/kube-api-access-kcqc6:{mountpoint:/var/lib/kubelet/pods/d417b861-3a8a-4889-8a97-3a03132e8623/volumes/kubernetes.io~projected/kube-api-access-kcqc6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d417b861-3a8a-4889-8a97-3a03132e8623/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d417b861-3a8a-4889-8a97-3a03132e8623/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea8500af-fe54-4c1e-983d-7a7b82d866b3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ea8500af-fe54-4c1e-983d-7a7b82d866b3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/990b1921fc8988a9b74f34436ef5dd3287c67d1721d0ef2492b6e9998acb648c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/cf8f51759fd3ff200c0733d8430e1b987f01252b779df428b3644a43c3b6678a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e555b47a274de5e51ef4cf037f80df7238a63bedca57d5c7db72a528a5459960/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/fbdce8fdae9b30916e7eddbb527b4285e659447ce5a1c92ae1e339151ef425e9/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/15d2949ff5f992a88d0be0d9867045ac4c9a3ab16562f2a9b13a73949e824a63/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/43789a8620551a0e6f34f12258526a5e83c6a637b71432e19c94aa35dc0db15b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c53fc15dac67bd91725825ebd770256837e392b5930b9f9be50939b8f705b5cd/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/de95a9268e643bbec9f213186bfd34132e0f7d06defaf653e43ff5d2383b4e94/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0d2b70baea4c206fc296b8bf250ccbec602a9e76034b64687235208e999a63e2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/55a1137410f11b6ce23c5d4aea975d2e6737c53df318cb1078a597932a9ce44b/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b85b4e45addd18980a24e81865eed66466e78d7b35af2b90a08f25184f16d442/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/be41d63ea8bb420bd6910a176ed54a8956e5bc6d192c850e1ef04c9525463abd/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/99135b690c592d5f5a59cf2e1a82d1c9bfe356d19232d3ec12dc1d4e1e400a83/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5203189899c04cf0aef21d042096193a23a2ab51cd972691021af78246e65b64/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/dd3b91b80ec96e76e12caf25c67116f9d234481f25d3121c42e2a097296e9d15/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/21045e72e633a547a51a714f5326ba14def2c793f038efb0744dfa6ef279997c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9d4eb6827fbd619be8326a009345923164557dbee818bccc4cd8ffd29e735011/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/247eef6e2429529726d6469b914cb43c3dcd277dd9210e3a2f0a04ca04ac149b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ce19792455f8f7f4e5202c38083e9ed4544ac02fba18f1afd81c14d075f67481/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2cdad399ee7f56f90660bb80d12226d54d160a3a6c7868909d644fb5f8299feb/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2e15859dad957ba651ade50f242df97bd71cf3c5188a044726be6c43713e9322/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/0aafa6aca92ebf053aef8bf8f540b78e76739c26b03c778315e72fb33677c1f8/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/a7e555429203d27814257b8a94c3a63768e38e94d95c46e30b358372a8b34636/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/953a9b7d4f5290979ece293f0e87fb08bd2dc3a707db0dde4d9bb691f3425793/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/91a1744f2f68eac37156da449d59e7eb1c7e8ca6c0fa0fb91efbe997f6dc2cab/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/c56bf86149e02c84713034bb6082fa617dd47b196e61086094d718563d48ac25/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/3d39858e084c7490dff91bc2b41529ad8aee2af05698f91b7c995b6b2da704d3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2cf8fd86699138ad3d167c30eabbf33df4834593299511f449bc68b4dde8ade3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9ac6f12de55ea8075bfaad435f1b4813c39a72aaaefe389b1cb19c5a1b1e1db8/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/9e99854fa5b2964daf29ef8314b07600c200acfe46abde8049bc12458eb4f18c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/0ae6b1f157c5eca3a2494fa49b56f79ffcf610e8e175ecaf090c8fb816943352/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/db9bf3fbded5f6fb012ba0896a68cff0d522530669a1418a87b6b0a93d92afac/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/44c88369d30e554db80a042b6cc8a055359b2b2d8e7172377fdc4715686eb4fe/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/04d6840935d922169fc004134bcb1c1ed7332bb7b9f1edfa397ce3f336c5f563/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/6c3392f0f76c3907c8e3fa0e2a94550a81deb0ee3be84f028b8aca0de2e78a9e/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 02 17:40:02.628733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.624614 4310 manager.go:217] Machine: {Timestamp:2026-11-02 17:40:02.623804341 +0000 UTC m=+0.066886981 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097347 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23e029fb56654d512d7ab1b3dae068 SystemUUID:ec23e029-fb56-654d-512d-7ab1b3dae068 BootID:e4981161-1f49-406d-b19d-dad229db1435 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/a3b7e801-25bc-4f6e-be57-f3131d2ce39c/volumes/kubernetes.io~projected/kube-api-access-bwgpc DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9edee41d-f2cc-404e-ae90-33dd5312d523/volumes/kubernetes.io~projected/kube-api-access-r8pkt DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9762673b-765b-4d8c-8791-ee16e740c1b3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3d7515ca0e1a3c98d3379022398065f010623807d452a28d1550585dedfe092/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/128194c44a764a57564fb9be1145f552a6eb7f8d040141336abd66f6679d0a01/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/688f509d6c74b21ba8392f7aea20f7e5768fdf28f9f5f8ac380840e4f0716dbc/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/61d00064-21c5-44b2-a467-18d93e752363/volumes/kubernetes.io~projected/kube-api-access-9swdz DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e/volumes/kubernetes.io~projected/kube-api-access-fbfg9 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65ff996d599a7608f7fe9d97fa4e09ee14011e646a4240e9c06654e0aa6f23db/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4307ecfe-be36-4891-9c9d-4c649e9aca70/volumes/kubernetes.io~projected/kube-api-access-8rtjz DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea8500af-fe54-4c1e-983d-7a7b82d866b3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d417b861-3a8a-4889-8a97-3a03132e8623/volumes/kubernetes.io~projected/kube-api-access-kcqc6 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/728f0b46110f27713f0ca1d3a806dc39fa67ac7702e9ee38f172c20af05a5264/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95cc5486-946f-4ada-be5d-84fc0068d031/volumes/kubernetes.io~projected/kube-api-access-q6nsz DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4b3bee0a5dafecb36831bd35a2f800a2cf529c89088c55c27e447ed184570b3/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb1271a6e0a5408d6d886fbeb865b4b9275b44bdb4ed32570506aa64b1920993/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b54e22c2a99faf8975976fa32ca0768be32ef355499d057800cb384fdb520f86/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b63aff75-00c5-4a5c-b9d3-678e6ed4465f/volumes/kubernetes.io~projected/kube-api-access-qsrdf DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9762673b-765b-4d8c-8791-ee16e740c1b3/volumes/kubernetes.io~projected/kube-api-access-znq7v DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d417b861-3a8a-4889-8a97-3a03132e8623/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d417b861-3a8a-4889-8a97-3a03132e8623/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9762673b-765b-4d8c-8791-ee16e740c1b3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53afcb4f0b6a07dad464bc1959e6a377e02b71721178be3604d11ec340930bd4/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9586f27793af460b023b7e5c1741f9b7cd0d229c3cd46cc5e42631b3bab2bec/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e/volumes/kubernetes.io~projected/kube-api-access-kln72 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d4854f8417a737d3de81577301ce23da7e44e0f890424ff66a81342462cfb98/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84ead3af633f8a0a68a58e26934899eb447ced88b7d61dc43a6a74e4d844f935/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:688f509d6c74b21 MacAddress:62:35:c7:fd:a6:95 Speed:10000 Mtu:8901} {Name:b54e22c2a99faf8 MacAddress:62:43:fc:51:75:29 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f3:93:e2:fd Speed:0 Mtu:9001} {Name:br-int MacAddress:6e:de:e1:b5:3a:36 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f3:93:e2:fd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:1a:2b:30:f4:cc:34 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ae:97:af:ea:ed:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:40:02.628733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625006 4310 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:40:02.628733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625065 4310 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:40:02.628733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625361 4310 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625393 4310 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-9.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625519 4310 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625527 4310 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625538 4310 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625557 4310 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625658 4310 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625815 4310 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625917 4310 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625951 4310 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625961 4310 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625970 4310 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625980 4310 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.625988 4310 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.626667 4310 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.626676 4310 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.627217 4310 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629019 4310 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:40:02.629045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629028 4310 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629131 4310 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629309 4310 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629335 4310 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629424 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629436 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629443 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629449 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629456 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629462 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629469 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629474 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629485 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629493 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629502 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:40:02.629524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629512 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:40:02.629999 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629542 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:40:02.629999 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.629551 4310 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:40:02.633335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.633323 4310 server.go:1263] "Started kubelet" Nov 02 17:40:02.634268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.633588 4310 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:40:02.634268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.633680 4310 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:40:02.634268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.633836 4310 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:40:02.634268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.634194 4310 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:40:02.633893 ip-10-0-135-9 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:40:02.637022 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.636999 4310 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:40:02.637595 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.637514 4310 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:40:02.637705 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.637644 4310 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:34:25 +0000 UTC" deadline="2028-07-03 12:46:00.833547516 +0000 UTC" Nov 02 17:40:02.637705 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.637679 4310 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14611h5m58.195872065s" Nov 02 17:40:02.637885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.637845 4310 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:40:02.637885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.637855 4310 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:40:02.638013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.637975 4310 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:40:02.638211 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.638176 4310 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:40:02.638852 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.638697 4310 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:40:02.639455 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639333 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3b7e801-25bc-4f6e-be57-f3131d2ce39c" volumeName="kubernetes.io/projected/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-kube-api-access-bwgpc" seLinuxMountContext="" Nov 02 17:40:02.639455 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639385 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b63aff75-00c5-4a5c-b9d3-678e6ed4465f" volumeName="kubernetes.io/projected/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-kube-api-access-qsrdf" seLinuxMountContext="" Nov 02 17:40:02.639455 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639412 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d417b861-3a8a-4889-8a97-3a03132e8623" volumeName="kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-config" seLinuxMountContext="" Nov 02 17:40:02.639455 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639431 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d417b861-3a8a-4889-8a97-3a03132e8623" volumeName="kubernetes.io/projected/d417b861-3a8a-4889-8a97-3a03132e8623-kube-api-access-kcqc6" seLinuxMountContext="" Nov 02 17:40:02.639686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639448 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" volumeName="kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 17:40:02.639686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639564 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e" volumeName="kubernetes.io/projected/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kube-api-access-fbfg9" seLinuxMountContext="" Nov 02 17:40:02.639686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639582 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4307ecfe-be36-4891-9c9d-4c649e9aca70" volumeName="kubernetes.io/empty-dir/4307ecfe-be36-4891-9c9d-4c649e9aca70-tmp-dir" seLinuxMountContext="" Nov 02 17:40:02.639686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639606 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95cc5486-946f-4ada-be5d-84fc0068d031" volumeName="kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz" seLinuxMountContext="" Nov 02 17:40:02.639686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639628 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9762673b-765b-4d8c-8791-ee16e740c1b3" volumeName="kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-tuned" seLinuxMountContext="" Nov 02 17:40:02.639686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639652 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" volumeName="kubernetes.io/projected/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-kube-api-access-kln72" seLinuxMountContext="" Nov 02 17:40:02.639686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639670 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d417b861-3a8a-4889-8a97-3a03132e8623" volumeName="kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-env-overrides" seLinuxMountContext="" Nov 02 17:40:02.639686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639687 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13" volumeName="kubernetes.io/configmap/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-konnectivity-ca" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639704 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61d00064-21c5-44b2-a467-18d93e752363" volumeName="kubernetes.io/projected/61d00064-21c5-44b2-a467-18d93e752363-kube-api-access-9swdz" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639720 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9edee41d-f2cc-404e-ae90-33dd5312d523" volumeName="kubernetes.io/projected/9edee41d-f2cc-404e-ae90-33dd5312d523-kube-api-access-r8pkt" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639743 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b63aff75-00c5-4a5c-b9d3-678e6ed4465f" volumeName="kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cni-binary-copy" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639755 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d417b861-3a8a-4889-8a97-3a03132e8623" volumeName="kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-script-lib" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639772 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4307ecfe-be36-4891-9c9d-4c649e9aca70" volumeName="kubernetes.io/projected/4307ecfe-be36-4891-9c9d-4c649e9aca70-kube-api-access-8rtjz" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639785 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13" volumeName="kubernetes.io/secret/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-agent-certs" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639797 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9762673b-765b-4d8c-8791-ee16e740c1b3" volumeName="kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-tmp" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639815 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9762673b-765b-4d8c-8791-ee16e740c1b3" volumeName="kubernetes.io/projected/9762673b-765b-4d8c-8791-ee16e740c1b3-kube-api-access-znq7v" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639827 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3b7e801-25bc-4f6e-be57-f3131d2ce39c" volumeName="kubernetes.io/configmap/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-serviceca" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639844 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b63aff75-00c5-4a5c-b9d3-678e6ed4465f" volumeName="kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-daemon-config" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639856 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d417b861-3a8a-4889-8a97-3a03132e8623" volumeName="kubernetes.io/secret/d417b861-3a8a-4889-8a97-3a03132e8623-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639870 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea8500af-fe54-4c1e-983d-7a7b82d866b3" volumeName="kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639894 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61d00064-21c5-44b2-a467-18d93e752363" volumeName="kubernetes.io/configmap/61d00064-21c5-44b2-a467-18d93e752363-iptables-alerter-script" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639908 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" volumeName="kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-binary-copy" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639925 4310 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" volumeName="kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639938 4310 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639945 4310 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.639997 4310 factory.go:153] Registering CRI-O factory Nov 02 17:40:02.640024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.640018 4310 factory.go:223] Registration of the crio container factory successfully Nov 02 17:40:02.640858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.640093 4310 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:40:02.640858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.640104 4310 factory.go:55] Registering systemd factory Nov 02 17:40:02.640858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.640112 4310 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:40:02.640858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.640138 4310 factory.go:103] Registering Raw factory Nov 02 17:40:02.640858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.640154 4310 manager.go:1199] Started watching for new ooms in manager Nov 02 17:40:02.641063 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.641056 4310 manager.go:319] Starting recovery of all containers Nov 02 17:40:02.644324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.642858 4310 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:40:02.644324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.643557 4310 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:40:02.644324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.644234 4310 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 17:40:02.664189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.664126 4310 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:40:02.665660 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.665637 4310 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:40:02.665660 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.665660 4310 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:40:02.665825 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.665682 4310 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:40:02.665825 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:40:02.665722 4310 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 17:40:02.669422 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.669384 4310 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:40:02.687340 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.687311 4310 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="43ce40ba52fa12694bff473c36eb577cb7bdbcbfbb21a52eda21042fd0b8d21b" exitCode=0 Nov 02 17:40:02.687340 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.687335 4310 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="6db777e4d072fb23d30426d7e201e6979043b84fd186bf73fc65bac93a8a7903" exitCode=0 Nov 02 17:40:02.687340 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.687341 4310 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="06dbb2d07a9415751e290d09e5ce1cb079bfdf4aac18e6b2b0f4a0f849361d61" exitCode=0 Nov 02 17:40:02.687513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.687349 4310 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="978781b604e8c2dd6dad49af9e2d9cb86fb42d2322f95bce919261e70e0a445f" exitCode=0 Nov 02 17:40:02.687513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.687358 4310 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="bf9a1e767eaa102a72c60c391b1b6307fe7cb5a23d7d77b05c1da85d17148698" exitCode=0 Nov 02 17:40:02.687513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.687365 4310 generic.go:356] "Generic (PLEG): container finished" podID="9bbe9efd-9dab-4adf-a72f-e5ce853fb19e" containerID="ad0d0893b9bdec64f80270e29e16a6a35d9c346293bf9be884af29574abdfb91" exitCode=0 Nov 02 17:40:02.689096 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.689058 4310 generic.go:356] "Generic (PLEG): container finished" podID="3b8fb63764518311609bba29a4e6d416" containerID="5a2ab06b544a5542188bfd0ad8564f2fec1bf331ceccb794f41ad056b0195861" exitCode=0 Nov 02 17:40:02.698637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.698620 4310 manager.go:324] Recovery completed Nov 02 17:40:02.712188 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712166 4310 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:40:02.712188 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712182 4310 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:40:02.712297 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712201 4310 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:40:02.712355 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712340 4310 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 17:40:02.712392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712357 4310 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 17:40:02.712392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712379 4310 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 17:40:02.712392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712386 4310 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 17:40:02.712471 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712393 4310 policy_none.go:50] "Start" Nov 02 17:40:02.712471 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712404 4310 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:40:02.712471 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712417 4310 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:40:02.712566 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712555 4310 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 17:40:02.712599 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712566 4310 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 17:40:02.712599 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.712572 4310 policy_none.go:44] "Start" Nov 02 17:40:02.716391 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.716368 4310 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:40:02.718763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.718750 4310 manager.go:341] "Starting Device Plugin manager" Nov 02 17:40:02.718827 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:40:02.718774 4310 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:40:02.718827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.718783 4310 server.go:87] "Starting device plugin registration server" Nov 02 17:40:02.719765 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.719749 4310 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:40:02.719836 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.719767 4310 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:40:02.719894 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.719874 4310 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:40:02.720008 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.719979 4310 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:40:02.720068 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.720024 4310 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:40:02.720068 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.720034 4310 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:40:02.720187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.720071 4310 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:40:02.720006868Z","UUID":"d56d4607-8e24-431a-aaab-cb18dfcae25c","Handler":null,"Name":"","Endpoint":""} Nov 02 17:40:02.721041 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:40:02.721021 4310 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:40:02.721293 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.721276 4310 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:40:02.721392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.721302 4310 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:40:02.766161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.766047 4310 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal"] Nov 02 17:40:02.766567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.766532 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" event={"ID":"06cff5a60959fbf288446179900e7d21","Type":"ContainerStarted","Data":"d847a36e01dcca44e02761f4217fd1f7a2657e5faea23383b1bf45b52a719b26"} Nov 02 17:40:02.766656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.766572 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" event={"ID":"06cff5a60959fbf288446179900e7d21","Type":"ContainerStarted","Data":"b9586f27793af460b023b7e5c1741f9b7cd0d229c3cd46cc5e42631b3bab2bec"} Nov 02 17:40:02.766656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.766619 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" event={"ID":"3b8fb63764518311609bba29a4e6d416","Type":"ContainerStarted","Data":"ad829fb1d1dd376ccdd57a0e3b7cde1a91ae253c6640fe0969e62b5cecebfc38"} Nov 02 17:40:02.766656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.766632 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" event={"ID":"3b8fb63764518311609bba29a4e6d416","Type":"ContainerDied","Data":"5a2ab06b544a5542188bfd0ad8564f2fec1bf331ceccb794f41ad056b0195861"} Nov 02 17:40:02.766656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.766644 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" event={"ID":"3b8fb63764518311609bba29a4e6d416","Type":"ContainerStarted","Data":"b4b3bee0a5dafecb36831bd35a2f800a2cf529c89088c55c27e447ed184570b3"} Nov 02 17:40:02.766844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.766655 4310 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.766844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.766812 4310 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.778452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.778437 4310 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:40:02.778526 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:40:02.778475 4310 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.778728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.778714 4310 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:40:02.778788 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:40:02.778753 4310 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.819865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.819832 4310 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:40:02.821115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.821098 4310 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:40:02.821224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.821132 4310 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:40:02.821224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.821147 4310 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 17:40:02.821224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.821175 4310 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.835462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.835441 4310 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.835519 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.835508 4310 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.845638 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.845612 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.845760 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.845643 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.845760 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.845668 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06cff5a60959fbf288446179900e7d21-config\") pod \"kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"06cff5a60959fbf288446179900e7d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.945981 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.945943 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.945981 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.945891 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.946217 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.946005 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.946217 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.946021 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06cff5a60959fbf288446179900e7d21-config\") pod \"kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"06cff5a60959fbf288446179900e7d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.946217 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.946048 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06cff5a60959fbf288446179900e7d21-config\") pod \"kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"06cff5a60959fbf288446179900e7d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:02.946217 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:02.946116 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b8fb63764518311609bba29a4e6d416-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" (UID: \"3b8fb63764518311609bba29a4e6d416\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:03.627119 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.627064 4310 apiserver.go:52] "Watching apiserver" Nov 02 17:40:03.633646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.633630 4310 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:40:03.634011 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.633994 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-8gq7s","openshift-network-diagnostics/network-check-target-x4786","openshift-network-operator/iptables-alerter-z7lsw","openshift-ovn-kubernetes/ovnkube-node-s8hhd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw","openshift-dns/node-resolver-59bfz","openshift-multus/network-metrics-daemon-wtm26","kube-system/konnectivity-agent-586q5","kube-system/kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-l8c9g","kube-system/global-pull-secret-syncer-pbcbk","openshift-image-registry/node-ca-2crqk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal","openshift-multus/multus-ksps4"] Nov 02 17:40:03.634445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.634429 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:40:03.638293 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.638268 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:40:03.638405 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.638313 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:40:03.639200 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.639175 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mkftb\"" Nov 02 17:40:03.639494 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.639468 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:40:03.639620 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.639552 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:40:03.639688 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.639639 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:40:03.639742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.639724 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7mg2x\"" Nov 02 17:40:03.639909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.639885 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:40:03.640232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.640214 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:40:03.640759 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.640734 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:40:03.640759 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.640739 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:40:03.640907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.640835 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:40:03.641048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.641029 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:40:03.641284 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.641261 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rjwn5\"" Nov 02 17:40:03.641573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.641557 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jk79z\"" Nov 02 17:40:03.641925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.641911 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:40:03.642099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642063 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b2827\"" Nov 02 17:40:03.642164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642097 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zqcnj\"" Nov 02 17:40:03.642164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642101 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:40:03.642164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642125 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:40:03.642164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642136 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:40:03.642341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642171 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:40:03.642341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642176 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-kkqh6\"" Nov 02 17:40:03.642341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642172 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:40:03.642341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642183 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:40:03.642341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642215 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:40:03.642341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642220 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:40:03.642341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642251 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:40:03.642341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642256 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4dl6t\"" Nov 02 17:40:03.642341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642257 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:40:03.642964 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.642950 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:40:03.643127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.643116 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:40:03.643243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.643231 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6x7d8\"" Nov 02 17:40:03.643724 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.643715 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:40:03.643759 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.643750 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:40:03.643897 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.643888 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4zvsl\"" Nov 02 17:40:03.644933 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.644913 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:40:03.645034 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.644942 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:40:03.645034 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.644951 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:40:03.645034 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.644955 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:40:03.645338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.645324 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzjcr\"" Nov 02 17:40:03.645456 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.645444 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:40:03.645586 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.645572 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpgl9\"" Nov 02 17:40:03.650365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650344 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rtjz\" (UniqueName: \"kubernetes.io/projected/4307ecfe-be36-4891-9c9d-4c649e9aca70-kube-api-access-8rtjz\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:40:03.650446 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650369 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.650504 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650488 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-bin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.650546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650517 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-daemon-config\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.650546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650537 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-multus\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.650611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650560 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-hostroot\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.650646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650614 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-conf-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.650646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650630 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-etc-kubernetes\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.650779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650650 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-dbus\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:40:03.650779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650685 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.650779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650696 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-systemd-units\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.650779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650717 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swdz\" (UniqueName: \"kubernetes.io/projected/61d00064-21c5-44b2-a467-18d93e752363-kube-api-access-9swdz\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:40:03.650927 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650778 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrdf\" (UniqueName: \"kubernetes.io/projected/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-kube-api-access-qsrdf\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.650927 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650884 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-daemon-config\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.650927 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650915 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-ovn\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651093 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650960 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-device-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.651093 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.650995 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61d00064-21c5-44b2-a467-18d93e752363-iptables-alerter-script\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:40:03.651093 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651054 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-config\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651233 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651138 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cni-binary-copy\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.651233 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651156 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-etc-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651233 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651171 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-agent-certs\") pod \"konnectivity-agent-586q5\" (UID: \"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13\") " pod="kube-system/konnectivity-agent-586q5" Nov 02 17:40:03.651233 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651187 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.651233 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651205 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-k8s-cni-cncf-io\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.651435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651239 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-var-lib-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651260 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-config\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651273 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651298 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-netd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651322 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61d00064-21c5-44b2-a467-18d93e752363-iptables-alerter-script\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:40:03.651435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651326 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-script-lib\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651370 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcqc6\" (UniqueName: \"kubernetes.io/projected/d417b861-3a8a-4889-8a97-3a03132e8623-kube-api-access-kcqc6\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651418 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cni-binary-copy\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.651435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651435 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-system-cni-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.651703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651508 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-agent-certs\") pod \"konnectivity-agent-586q5\" (UID: \"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13\") " pod="kube-system/konnectivity-agent-586q5" Nov 02 17:40:03.651703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651527 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-ovnkube-script-lib\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651550 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-socket-dir-parent\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.651703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651574 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-slash\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651607 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-systemd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651631 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-env-overrides\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651670 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-sys-fs\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.651879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651702 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nsz\" (UniqueName: \"kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz\") pod \"network-check-target-x4786\" (UID: \"95cc5486-946f-4ada-be5d-84fc0068d031\") " pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:40:03.651879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651749 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-netns\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651766 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.651879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651784 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d417b861-3a8a-4889-8a97-3a03132e8623-env-overrides\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.651879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651792 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-registration-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.651879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651829 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-konnectivity-ca\") pod \"konnectivity-agent-586q5\" (UID: \"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13\") " pod="kube-system/konnectivity-agent-586q5" Nov 02 17:40:03.652035 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651878 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cnibin\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.652035 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651937 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-kubelet\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.652035 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651954 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-node-log\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.652035 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651977 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-bin\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.652035 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.651974 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13-konnectivity-ca\") pod \"konnectivity-agent-586q5\" (UID: \"5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13\") " pod="kube-system/konnectivity-agent-586q5" Nov 02 17:40:03.652035 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652017 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d417b861-3a8a-4889-8a97-3a03132e8623-ovn-node-metrics-cert\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.652209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652040 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-socket-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.652209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652147 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbfg9\" (UniqueName: \"kubernetes.io/projected/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kube-api-access-fbfg9\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.652209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652172 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-kubelet-config\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:40:03.652209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652188 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-multus-certs\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.652209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652202 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61d00064-21c5-44b2-a467-18d93e752363-host-slash\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:40:03.652339 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652223 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4307ecfe-be36-4891-9c9d-4c649e9aca70-tmp-dir\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:40:03.652370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652343 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kln72\" (UniqueName: \"kubernetes.io/projected/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-kube-api-access-kln72\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.652370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652356 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4307ecfe-be36-4891-9c9d-4c649e9aca70-tmp-dir\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:40:03.652436 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652414 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-log-socket\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.652468 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652438 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d417b861-3a8a-4889-8a97-3a03132e8623-ovn-node-metrics-cert\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.652541 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652525 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.652583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652568 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-system-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.652611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652597 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cnibin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.652639 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652624 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-kubelet\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.652672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652642 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.652715 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652669 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.652745 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652729 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8pkt\" (UniqueName: \"kubernetes.io/projected/9edee41d-f2cc-404e-ae90-33dd5312d523-kube-api-access-r8pkt\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:40:03.652775 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652767 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4307ecfe-be36-4891-9c9d-4c649e9aca70-hosts-file\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:40:03.652819 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652804 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-binary-copy\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.652851 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652841 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:40:03.652884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652866 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-netns\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.652948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652915 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-etc-selinux\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.652996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652945 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.652996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652963 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:40:03.652996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.652995 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.653152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.653039 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cni-binary-copy\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.653272 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.653254 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8500af-fe54-4c1e-983d-7a7b82d866b3-original-pull-secret\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:40:03.656559 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.656542 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rtjz\" (UniqueName: \"kubernetes.io/projected/4307ecfe-be36-4891-9c9d-4c649e9aca70-kube-api-access-8rtjz\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:40:03.657671 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.657648 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrdf\" (UniqueName: \"kubernetes.io/projected/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-kube-api-access-qsrdf\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.657750 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.657690 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swdz\" (UniqueName: \"kubernetes.io/projected/61d00064-21c5-44b2-a467-18d93e752363-kube-api-access-9swdz\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:40:03.658158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.658132 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcqc6\" (UniqueName: \"kubernetes.io/projected/d417b861-3a8a-4889-8a97-3a03132e8623-kube-api-access-kcqc6\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.658205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.658190 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6nsz\" (UniqueName: \"kubernetes.io/projected/95cc5486-946f-4ada-be5d-84fc0068d031-kube-api-access-q6nsz\") pod \"network-check-target-x4786\" (UID: \"95cc5486-946f-4ada-be5d-84fc0068d031\") " pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:40:03.660858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.660842 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbfg9\" (UniqueName: \"kubernetes.io/projected/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kube-api-access-fbfg9\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.660858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.660851 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kln72\" (UniqueName: \"kubernetes.io/projected/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-kube-api-access-kln72\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.661126 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.661111 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8pkt\" (UniqueName: \"kubernetes.io/projected/9edee41d-f2cc-404e-ae90-33dd5312d523-kube-api-access-r8pkt\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:40:03.696883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.696853 4310 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:03.707201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.707173 4310 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:40:03.707316 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:40:03.707218 4310 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-9.us-east-2.compute.internal" Nov 02 17:40:03.738762 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.738727 4310 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:40:03.753956 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.753923 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-device-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.753956 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.753960 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-host\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.754197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.753983 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-tuned\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.754197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754006 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-etc-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754049 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-device-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.754197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754111 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-etc-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754135 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-modprobe-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.754197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754153 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-tuned\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.754197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754175 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-tmp\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.754197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754196 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-serviceca\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754224 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754240 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-k8s-cni-cncf-io\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754256 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-var-lib-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754271 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754284 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-netd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754287 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9762673b-765b-4d8c-8791-ee16e740c1b3-tmp\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754306 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-system-cni-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754316 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-k8s-cni-cncf-io\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754330 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-socket-dir-parent\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754347 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-var-lib-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754354 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-slash\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754371 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-netd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754371 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-system-cni-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754357 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754379 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-systemd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754408 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-socket-dir-parent\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754416 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-systemd\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754419 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-slash\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754428 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysconfig\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754389 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754449 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znq7v\" (UniqueName: \"kubernetes.io/projected/9762673b-765b-4d8c-8791-ee16e740c1b3-kube-api-access-znq7v\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754450 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-serviceca\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754491 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-sys-fs\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.754636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754629 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-netns\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754646 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-sys-fs\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754650 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-sys\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754686 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-run-netns\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754699 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754717 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-registration-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754733 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cnibin\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754748 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-kubelet\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754770 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-node-log\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754793 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-bin\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754796 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-cnibin\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754804 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754796 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-registration-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754812 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-kubelet\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754831 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-cni-bin\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754839 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-node-log\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754874 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-socket-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754908 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-socket-dir\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754918 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-kubelet-config\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754951 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-run\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754956 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-kubelet-config\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.754976 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-multus-certs\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755005 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61d00064-21c5-44b2-a467-18d93e752363-host-slash\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755036 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-multus-certs\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755059 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61d00064-21c5-44b2-a467-18d93e752363-host-slash\") pod \"iptables-alerter-z7lsw\" (UID: \"61d00064-21c5-44b2-a467-18d93e752363\") " pod="openshift-network-operator/iptables-alerter-z7lsw" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755112 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-log-socket\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755128 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755143 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-system-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755164 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cnibin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755179 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-kubelet\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755198 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-log-socket\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755204 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbe9efd-9dab-4adf-a72f-e5ce853fb19e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l8c9g\" (UID: \"9bbe9efd-9dab-4adf-a72f-e5ce853fb19e\") " pod="openshift-multus/multus-additional-cni-plugins-l8c9g" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755214 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755225 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-cnibin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755225 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-system-cni-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755253 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-kubelet\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755263 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755275 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-kubernetes\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755290 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755307 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-lib-modules\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755375 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-host\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755404 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgpc\" (UniqueName: \"kubernetes.io/projected/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-kube-api-access-bwgpc\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755420 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4307ecfe-be36-4891-9c9d-4c649e9aca70-hosts-file\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755468 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755478 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4307ecfe-be36-4891-9c9d-4c649e9aca70-hosts-file\") pod \"node-resolver-59bfz\" (UID: \"4307ecfe-be36-4891-9c9d-4c649e9aca70\") " pod="openshift-dns/node-resolver-59bfz" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755502 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-netns\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755537 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-etc-selinux\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.755532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755562 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-var-lib-kubelet\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755588 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755600 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-run-netns\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755608 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e-etc-selinux\") pod \"aws-ebs-csi-driver-node-77wjw\" (UID: \"1d7cdb9f-dd7b-4b23-bf0a-0d23fb6e382e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-77wjw" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755612 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-conf\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755658 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-systemd\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755664 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-openvswitch\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755706 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-bin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755793 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-multus\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755801 4310 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755820 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-bin\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755837 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-hostroot\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755884 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-host-var-lib-cni-multus\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755894 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-hostroot\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755889 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-conf-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755940 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-multus-conf-dir\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755952 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-etc-kubernetes\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.755981 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-dbus\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.756003 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-systemd-units\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.756030 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b63aff75-00c5-4a5c-b9d3-678e6ed4465f-etc-kubernetes\") pod \"multus-ksps4\" (UID: \"b63aff75-00c5-4a5c-b9d3-678e6ed4465f\") " pod="openshift-multus/multus-ksps4" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.756040 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-ovn\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.756070 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea8500af-fe54-4c1e-983d-7a7b82d866b3-dbus\") pod \"global-pull-secret-syncer-pbcbk\" (UID: \"ea8500af-fe54-4c1e-983d-7a7b82d866b3\") " pod="kube-system/global-pull-secret-syncer-pbcbk" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.756158 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-systemd-units\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.757003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.756189 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d417b861-3a8a-4889-8a97-3a03132e8623-run-ovn\") pod \"ovnkube-node-s8hhd\" (UID: \"d417b861-3a8a-4889-8a97-3a03132e8623\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:03.757828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.757812 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edee41d-f2cc-404e-ae90-33dd5312d523-metrics-certs\") pod \"network-metrics-daemon-wtm26\" (UID: \"9edee41d-f2cc-404e-ae90-33dd5312d523\") " pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:40:03.764379 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.764359 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znq7v\" (UniqueName: \"kubernetes.io/projected/9762673b-765b-4d8c-8791-ee16e740c1b3-kube-api-access-znq7v\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.764986 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.764968 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgpc\" (UniqueName: \"kubernetes.io/projected/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-kube-api-access-bwgpc\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:40:03.783568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.783536 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:03.856777 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856749 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysconfig\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.856882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856781 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-sys\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.856882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856799 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-run\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.856882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856821 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-kubernetes\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.856882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856845 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.856882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856866 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-lib-modules\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856888 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-host\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856890 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysconfig\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856907 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-kubernetes\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856911 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-run\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856913 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-var-lib-kubelet\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856888 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-sys\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856944 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3b7e801-25bc-4f6e-be57-f3131d2ce39c-host\") pod \"node-ca-2crqk\" (UID: \"a3b7e801-25bc-4f6e-be57-f3131d2ce39c\") " pod="openshift-image-registry/node-ca-2crqk" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856982 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-lib-modules\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.856990 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.857005 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-var-lib-kubelet\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.857023 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-conf\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.857041 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-systemd\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.857065 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-host\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.857095 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-modprobe-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.857106 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-systemd\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.857139 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-host\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.857173 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-modprobe-d\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.857567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.857176 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9762673b-765b-4d8c-8791-ee16e740c1b3-etc-sysctl-conf\") pod \"tuned-8gq7s\" (UID: \"9762673b-765b-4d8c-8791-ee16e740c1b3\") " pod="openshift-cluster-node-tuning-operator/tuned-8gq7s" Nov 02 17:40:03.934879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:03.934799 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtm26" Nov 02 17:40:04.105417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:04.105369 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:04.206700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:04.206624 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:04.241458 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:04.241428 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wtm26"] Nov 02 17:40:04.245007 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:04.244967 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edee41d_f2cc_404e_ae90_33dd5312d523.slice/crio-21103c4335229393b4153050c4dca21948934831eac85960ac4dd935f8a38f00 WatchSource:0}: Error finding container 21103c4335229393b4153050c4dca21948934831eac85960ac4dd935f8a38f00: Status 404 returned error can't find the container with id 21103c4335229393b4153050c4dca21948934831eac85960ac4dd935f8a38f00 Nov 02 17:40:04.247238 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:04.247221 4310 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:40:04.378450 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:04.378413 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:04.699366 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:04.699330 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtm26" event={"ID":"9edee41d-f2cc-404e-ae90-33dd5312d523","Type":"ContainerStarted","Data":"21103c4335229393b4153050c4dca21948934831eac85960ac4dd935f8a38f00"} Nov 02 17:40:06.706897 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:06.706859 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtm26" event={"ID":"9edee41d-f2cc-404e-ae90-33dd5312d523","Type":"ContainerStarted","Data":"0df00c332733f2d1d68dd131cbf9ade5a9a7a9b123a6f1e5f3ab8de3cbc315ce"} Nov 02 17:40:06.707389 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:06.706911 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtm26" event={"ID":"9edee41d-f2cc-404e-ae90-33dd5312d523","Type":"ContainerStarted","Data":"340234a4d8cf8e250a4a9b5e01e391a5cef5d04157f12eb7f8f148283e28c362"} Nov 02 17:40:06.956534 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:06.956501 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-586q5" Nov 02 17:40:07.925966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:07.925930 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:40:08.860856 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:08.860828 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:09.754427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:09.754397 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:09.754803 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:09.754601 4310 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:40:10.164916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:10.164825 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:10.260124 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:10.260097 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:11.568142 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.568103 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-57wqv"] Nov 02 17:40:11.569901 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.569884 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-57wqv" Nov 02 17:40:11.571944 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.571919 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2"] Nov 02 17:40:11.573444 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.573429 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2" Nov 02 17:40:11.577496 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.577472 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d94c8b5d9-8gspc"] Nov 02 17:40:11.579367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.579347 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zg6hp"] Nov 02 17:40:11.579496 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.579477 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.580992 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.580975 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.595061 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.595035 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf989\" (UniqueName: \"kubernetes.io/projected/a34b1f69-0317-462a-981b-51ae91f9e8b5-kube-api-access-wf989\") pod \"downloads-6dd789c8b7-57wqv\" (UID: \"a34b1f69-0317-462a-981b-51ae91f9e8b5\") " pod="openshift-console/downloads-6dd789c8b7-57wqv" Nov 02 17:40:11.595201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.595066 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/87d8bb05-337f-4eb5-b71f-f95f3b160ecb-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bvzg2\" (UID: \"87d8bb05-337f-4eb5-b71f-f95f3b160ecb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2" Nov 02 17:40:11.595252 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.595214 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a34b1f69-0317-462a-981b-51ae91f9e8b5-tmp\") pod \"downloads-6dd789c8b7-57wqv\" (UID: \"a34b1f69-0317-462a-981b-51ae91f9e8b5\") " pod="openshift-console/downloads-6dd789c8b7-57wqv" Nov 02 17:40:11.599372 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.599350 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 17:40:11.599536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.599520 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 17:40:11.599582 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.599563 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 17:40:11.600122 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.600105 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-n5j94\"" Nov 02 17:40:11.600198 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.600108 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-vk6xj\"" Nov 02 17:40:11.612054 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.612022 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-57wqv"] Nov 02 17:40:11.612442 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.612415 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2"] Nov 02 17:40:11.612541 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.612455 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zg6hp"] Nov 02 17:40:11.612864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.612840 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d94c8b5d9-8gspc"] Nov 02 17:40:11.621026 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.620983 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:40:11.621026 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.620999 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:40:11.621026 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.621018 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:40:11.621274 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.621008 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 17:40:11.621274 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.621117 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 17:40:11.621274 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.621161 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 17:40:11.650864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.650830 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7r6cv"] Nov 02 17:40:11.652935 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.652911 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7r6cv" Nov 02 17:40:11.666984 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.666960 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:40:11.666984 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.666981 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:40:11.675532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.675495 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tz2kg"] Nov 02 17:40:11.680772 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.680747 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c84fd649c-lxh8c"] Nov 02 17:40:11.681029 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.681007 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.682566 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.682549 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7r6cv"] Nov 02 17:40:11.682678 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.682627 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.693258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.693237 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:40:11.693258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.693256 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 17:40:11.693414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.693404 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:40:11.693475 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.693450 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gjv4t\"" Nov 02 17:40:11.693531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.693501 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7flsb\"" Nov 02 17:40:11.693585 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.693577 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 17:40:11.694398 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.694381 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-scp8s\"" Nov 02 17:40:11.694490 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.694408 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 17:40:11.694490 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.694419 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ntx78\"" Nov 02 17:40:11.694490 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.694434 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bs2ss\"" Nov 02 17:40:11.694490 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.694480 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 17:40:11.694654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.694548 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:40:11.694654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.694597 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 17:40:11.694654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.694609 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:40:11.695776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.695756 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5350eca9-94c7-4dc9-a571-ec16e055d94d-crio-socket\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.695833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.695793 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-oauth-serving-cert\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.695833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.695818 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/486c0825-8a27-4863-bbb7-27e11440fb83-cert\") pod \"ingress-canary-7r6cv\" (UID: \"486c0825-8a27-4863-bbb7-27e11440fb83\") " pod="openshift-ingress-canary/ingress-canary-7r6cv" Nov 02 17:40:11.695959 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.695942 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-serving-cert\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.696061 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.695973 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-oauth-config\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.696061 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696005 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d4vm\" (UniqueName: \"kubernetes.io/projected/486c0825-8a27-4863-bbb7-27e11440fb83-kube-api-access-6d4vm\") pod \"ingress-canary-7r6cv\" (UID: \"486c0825-8a27-4863-bbb7-27e11440fb83\") " pod="openshift-ingress-canary/ingress-canary-7r6cv" Nov 02 17:40:11.696203 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696181 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a34b1f69-0317-462a-981b-51ae91f9e8b5-tmp\") pod \"downloads-6dd789c8b7-57wqv\" (UID: \"a34b1f69-0317-462a-981b-51ae91f9e8b5\") " pod="openshift-console/downloads-6dd789c8b7-57wqv" Nov 02 17:40:11.696264 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696218 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xpns\" (UniqueName: \"kubernetes.io/projected/5350eca9-94c7-4dc9-a571-ec16e055d94d-kube-api-access-8xpns\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.696264 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696242 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-config\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.696366 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696282 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z48wn\" (UniqueName: \"kubernetes.io/projected/e324050f-2d8c-49fb-9763-0c1df35aab9a-kube-api-access-z48wn\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.696366 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696326 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5350eca9-94c7-4dc9-a571-ec16e055d94d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.696366 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696363 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf989\" (UniqueName: \"kubernetes.io/projected/a34b1f69-0317-462a-981b-51ae91f9e8b5-kube-api-access-wf989\") pod \"downloads-6dd789c8b7-57wqv\" (UID: \"a34b1f69-0317-462a-981b-51ae91f9e8b5\") " pod="openshift-console/downloads-6dd789c8b7-57wqv" Nov 02 17:40:11.696507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696407 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/87d8bb05-337f-4eb5-b71f-f95f3b160ecb-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bvzg2\" (UID: \"87d8bb05-337f-4eb5-b71f-f95f3b160ecb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2" Nov 02 17:40:11.696507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696475 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e324050f-2d8c-49fb-9763-0c1df35aab9a-tmp\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.696610 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696554 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-service-ca\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.696610 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696565 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a34b1f69-0317-462a-981b-51ae91f9e8b5-tmp\") pod \"downloads-6dd789c8b7-57wqv\" (UID: \"a34b1f69-0317-462a-981b-51ae91f9e8b5\") " pod="openshift-console/downloads-6dd789c8b7-57wqv" Nov 02 17:40:11.696610 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696575 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5350eca9-94c7-4dc9-a571-ec16e055d94d-data-volume\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.696742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.696648 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5350eca9-94c7-4dc9-a571-ec16e055d94d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.697442 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.697425 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 17:40:11.698826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.698810 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/87d8bb05-337f-4eb5-b71f-f95f3b160ecb-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bvzg2\" (UID: \"87d8bb05-337f-4eb5-b71f-f95f3b160ecb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2" Nov 02 17:40:11.705390 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.705367 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tz2kg"] Nov 02 17:40:11.713003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.712982 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c84fd649c-lxh8c"] Nov 02 17:40:11.722565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.722536 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf989\" (UniqueName: \"kubernetes.io/projected/a34b1f69-0317-462a-981b-51ae91f9e8b5-kube-api-access-wf989\") pod \"downloads-6dd789c8b7-57wqv\" (UID: \"a34b1f69-0317-462a-981b-51ae91f9e8b5\") " pod="openshift-console/downloads-6dd789c8b7-57wqv" Nov 02 17:40:11.725270 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.725232 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:11.797516 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.797488 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xpns\" (UniqueName: \"kubernetes.io/projected/5350eca9-94c7-4dc9-a571-ec16e055d94d-kube-api-access-8xpns\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.797683 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.797522 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af0994d-e5fb-4ef5-af83-3662154a29fd-bound-sa-token\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.797683 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.797542 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-config\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.797683 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.797561 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f27a1622-72d7-411f-baba-028b6c2f94ee-metrics-tls\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.797844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.797740 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e324050f-2d8c-49fb-9763-0c1df35aab9a-tmp\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.797844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.797771 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-service-ca\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.797844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.797797 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2af0994d-e5fb-4ef5-af83-3662154a29fd-trusted-ca\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.797844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.797829 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2af0994d-e5fb-4ef5-af83-3662154a29fd-image-registry-private-configuration\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.798071 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.797864 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pjcq\" (UniqueName: \"kubernetes.io/projected/2af0994d-e5fb-4ef5-af83-3662154a29fd-kube-api-access-2pjcq\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.798071 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.797984 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-serving-cert\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.798071 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798023 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2af0994d-e5fb-4ef5-af83-3662154a29fd-registry-tls\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.798071 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798048 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2af0994d-e5fb-4ef5-af83-3662154a29fd-installation-pull-secrets\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.798351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798095 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d4vm\" (UniqueName: \"kubernetes.io/projected/486c0825-8a27-4863-bbb7-27e11440fb83-kube-api-access-6d4vm\") pod \"ingress-canary-7r6cv\" (UID: \"486c0825-8a27-4863-bbb7-27e11440fb83\") " pod="openshift-ingress-canary/ingress-canary-7r6cv" Nov 02 17:40:11.798351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798214 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e324050f-2d8c-49fb-9763-0c1df35aab9a-tmp\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.798351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798293 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2af0994d-e5fb-4ef5-af83-3662154a29fd-registry-certificates\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.798808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798412 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z48wn\" (UniqueName: \"kubernetes.io/projected/e324050f-2d8c-49fb-9763-0c1df35aab9a-kube-api-access-z48wn\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.798808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798449 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f27a1622-72d7-411f-baba-028b6c2f94ee-config-volume\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.798808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798475 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzxdw\" (UniqueName: \"kubernetes.io/projected/f27a1622-72d7-411f-baba-028b6c2f94ee-kube-api-access-wzxdw\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.798808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798504 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-service-ca\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.798808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798525 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-config\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.798808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798507 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5350eca9-94c7-4dc9-a571-ec16e055d94d-data-volume\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.798808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798739 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2af0994d-e5fb-4ef5-af83-3662154a29fd-ca-trust-extracted\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.798808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798786 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5350eca9-94c7-4dc9-a571-ec16e055d94d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.798808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798805 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5350eca9-94c7-4dc9-a571-ec16e055d94d-crio-socket\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.799335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798821 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5350eca9-94c7-4dc9-a571-ec16e055d94d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.799335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798846 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f27a1622-72d7-411f-baba-028b6c2f94ee-tmp-dir\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.799335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798875 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-oauth-serving-cert\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.799335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798884 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5350eca9-94c7-4dc9-a571-ec16e055d94d-data-volume\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.799335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798899 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/486c0825-8a27-4863-bbb7-27e11440fb83-cert\") pod \"ingress-canary-7r6cv\" (UID: \"486c0825-8a27-4863-bbb7-27e11440fb83\") " pod="openshift-ingress-canary/ingress-canary-7r6cv" Nov 02 17:40:11.799335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798900 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5350eca9-94c7-4dc9-a571-ec16e055d94d-crio-socket\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.799335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.798946 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-oauth-config\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.799675 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.799410 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5350eca9-94c7-4dc9-a571-ec16e055d94d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.799675 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.799517 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-oauth-serving-cert\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.800987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.800968 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-serving-cert\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.801520 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.801498 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5350eca9-94c7-4dc9-a571-ec16e055d94d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.801722 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.801702 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-oauth-config\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.801780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.801722 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/486c0825-8a27-4863-bbb7-27e11440fb83-cert\") pod \"ingress-canary-7r6cv\" (UID: \"486c0825-8a27-4863-bbb7-27e11440fb83\") " pod="openshift-ingress-canary/ingress-canary-7r6cv" Nov 02 17:40:11.809945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.809919 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xpns\" (UniqueName: \"kubernetes.io/projected/5350eca9-94c7-4dc9-a571-ec16e055d94d-kube-api-access-8xpns\") pod \"insights-runtime-extractor-zg6hp\" (UID: \"5350eca9-94c7-4dc9-a571-ec16e055d94d\") " pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.810173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.810156 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d4vm\" (UniqueName: \"kubernetes.io/projected/486c0825-8a27-4863-bbb7-27e11440fb83-kube-api-access-6d4vm\") pod \"ingress-canary-7r6cv\" (UID: \"486c0825-8a27-4863-bbb7-27e11440fb83\") " pod="openshift-ingress-canary/ingress-canary-7r6cv" Nov 02 17:40:11.810244 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.810228 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z48wn\" (UniqueName: \"kubernetes.io/projected/e324050f-2d8c-49fb-9763-0c1df35aab9a-kube-api-access-z48wn\") pod \"console-7d94c8b5d9-8gspc\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.881883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.881801 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-57wqv" Nov 02 17:40:11.888275 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.888255 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2" Nov 02 17:40:11.894161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.894136 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:11.899932 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.899907 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2af0994d-e5fb-4ef5-af83-3662154a29fd-image-registry-private-configuration\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.900032 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.899945 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pjcq\" (UniqueName: \"kubernetes.io/projected/2af0994d-e5fb-4ef5-af83-3662154a29fd-kube-api-access-2pjcq\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.900032 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.899971 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2af0994d-e5fb-4ef5-af83-3662154a29fd-registry-tls\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.900032 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.899986 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2af0994d-e5fb-4ef5-af83-3662154a29fd-installation-pull-secrets\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.900215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.900062 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2af0994d-e5fb-4ef5-af83-3662154a29fd-registry-certificates\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.900215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.900131 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f27a1622-72d7-411f-baba-028b6c2f94ee-config-volume\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.900215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.900170 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxdw\" (UniqueName: \"kubernetes.io/projected/f27a1622-72d7-411f-baba-028b6c2f94ee-kube-api-access-wzxdw\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.901108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.900489 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zg6hp" Nov 02 17:40:11.901108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.900525 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2af0994d-e5fb-4ef5-af83-3662154a29fd-ca-trust-extracted\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.901108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.900558 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f27a1622-72d7-411f-baba-028b6c2f94ee-tmp-dir\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.901108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.900604 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af0994d-e5fb-4ef5-af83-3662154a29fd-bound-sa-token\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.901108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.900631 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f27a1622-72d7-411f-baba-028b6c2f94ee-metrics-tls\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.901108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.900657 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2af0994d-e5fb-4ef5-af83-3662154a29fd-trusted-ca\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.901108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.900967 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f27a1622-72d7-411f-baba-028b6c2f94ee-tmp-dir\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.901108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.901008 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f27a1622-72d7-411f-baba-028b6c2f94ee-config-volume\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.903006 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.902563 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2af0994d-e5fb-4ef5-af83-3662154a29fd-registry-certificates\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.903006 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.902719 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2af0994d-e5fb-4ef5-af83-3662154a29fd-ca-trust-extracted\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.903815 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.903754 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2af0994d-e5fb-4ef5-af83-3662154a29fd-installation-pull-secrets\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.903815 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.903772 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2af0994d-e5fb-4ef5-af83-3662154a29fd-image-registry-private-configuration\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.903972 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.903818 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2af0994d-e5fb-4ef5-af83-3662154a29fd-trusted-ca\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.904682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.904663 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f27a1622-72d7-411f-baba-028b6c2f94ee-metrics-tls\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.906046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.906022 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2af0994d-e5fb-4ef5-af83-3662154a29fd-registry-tls\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.913152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.910605 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxdw\" (UniqueName: \"kubernetes.io/projected/f27a1622-72d7-411f-baba-028b6c2f94ee-kube-api-access-wzxdw\") pod \"dns-default-tz2kg\" (UID: \"f27a1622-72d7-411f-baba-028b6c2f94ee\") " pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.913152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.912452 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pjcq\" (UniqueName: \"kubernetes.io/projected/2af0994d-e5fb-4ef5-af83-3662154a29fd-kube-api-access-2pjcq\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.913152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.913135 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af0994d-e5fb-4ef5-af83-3662154a29fd-bound-sa-token\") pod \"image-registry-c84fd649c-lxh8c\" (UID: \"2af0994d-e5fb-4ef5-af83-3662154a29fd\") " pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:11.931718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.931278 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:11.965176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.965150 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7r6cv" Nov 02 17:40:11.993533 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.993405 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:11.997996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:11.997969 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:12.106779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.106743 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:12.175660 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.175589 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-57wqv"] Nov 02 17:40:12.179385 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:12.179359 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34b1f69_0317_462a_981b_51ae91f9e8b5.slice/crio-e778693a6feb7a4b68ad9b2e70ab5418e5bcc43939a2821db2e5ff865663665d WatchSource:0}: Error finding container e778693a6feb7a4b68ad9b2e70ab5418e5bcc43939a2821db2e5ff865663665d: Status 404 returned error can't find the container with id e778693a6feb7a4b68ad9b2e70ab5418e5bcc43939a2821db2e5ff865663665d Nov 02 17:40:12.187898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.187874 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2"] Nov 02 17:40:12.190961 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:12.190932 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d8bb05_337f_4eb5_b71f_f95f3b160ecb.slice/crio-16f4a5b3de427112aa9f92689582c4ff0d59b3d4ecf45fb0a02525fd0a6f42e4 WatchSource:0}: Error finding container 16f4a5b3de427112aa9f92689582c4ff0d59b3d4ecf45fb0a02525fd0a6f42e4: Status 404 returned error can't find the container with id 16f4a5b3de427112aa9f92689582c4ff0d59b3d4ecf45fb0a02525fd0a6f42e4 Nov 02 17:40:12.200552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.200531 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d94c8b5d9-8gspc"] Nov 02 17:40:12.203296 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:12.203272 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode324050f_2d8c_49fb_9763_0c1df35aab9a.slice/crio-3ccdf295ded318746969984f48c311ebb3fb6780ba655cdfa3854237c4dabf2c WatchSource:0}: Error finding container 3ccdf295ded318746969984f48c311ebb3fb6780ba655cdfa3854237c4dabf2c: Status 404 returned error can't find the container with id 3ccdf295ded318746969984f48c311ebb3fb6780ba655cdfa3854237c4dabf2c Nov 02 17:40:12.207838 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.207820 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zg6hp"] Nov 02 17:40:12.211331 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:12.211306 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5350eca9_94c7_4dc9_a571_ec16e055d94d.slice/crio-e0f46a483ffbbaa8a2f31ee07fc3ad58b9d7349c06eeba8e2f305b80c965d6be WatchSource:0}: Error finding container e0f46a483ffbbaa8a2f31ee07fc3ad58b9d7349c06eeba8e2f305b80c965d6be: Status 404 returned error can't find the container with id e0f46a483ffbbaa8a2f31ee07fc3ad58b9d7349c06eeba8e2f305b80c965d6be Nov 02 17:40:12.249212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.249188 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7r6cv"] Nov 02 17:40:12.252627 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:12.252600 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486c0825_8a27_4863_bbb7_27e11440fb83.slice/crio-44e4dcb1b88694ed72b1830dfd3b6e3aaf77d7a8c3cf3e691d44d061a81f2c2e WatchSource:0}: Error finding container 44e4dcb1b88694ed72b1830dfd3b6e3aaf77d7a8c3cf3e691d44d061a81f2c2e: Status 404 returned error can't find the container with id 44e4dcb1b88694ed72b1830dfd3b6e3aaf77d7a8c3cf3e691d44d061a81f2c2e Nov 02 17:40:12.279022 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.278984 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tz2kg"] Nov 02 17:40:12.283835 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:12.283809 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27a1622_72d7_411f_baba_028b6c2f94ee.slice/crio-913ebe41d7745f22fc64613510004b8b18c0bfe7704cebe8b6306c37218dbb08 WatchSource:0}: Error finding container 913ebe41d7745f22fc64613510004b8b18c0bfe7704cebe8b6306c37218dbb08: Status 404 returned error can't find the container with id 913ebe41d7745f22fc64613510004b8b18c0bfe7704cebe8b6306c37218dbb08 Nov 02 17:40:12.304242 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.304213 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c84fd649c-lxh8c"] Nov 02 17:40:12.307649 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:12.307627 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af0994d_e5fb_4ef5_af83_3662154a29fd.slice/crio-81d203ff2d6034a31c7047971f78a29ea6e98ecbed36d90f985de9be83156aa6 WatchSource:0}: Error finding container 81d203ff2d6034a31c7047971f78a29ea6e98ecbed36d90f985de9be83156aa6: Status 404 returned error can't find the container with id 81d203ff2d6034a31c7047971f78a29ea6e98ecbed36d90f985de9be83156aa6 Nov 02 17:40:12.625447 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.625413 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:12.724674 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.724635 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-57wqv" event={"ID":"a34b1f69-0317-462a-981b-51ae91f9e8b5","Type":"ContainerStarted","Data":"e778693a6feb7a4b68ad9b2e70ab5418e5bcc43939a2821db2e5ff865663665d"} Nov 02 17:40:12.726567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.726505 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:12.728280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.728034 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" event={"ID":"2af0994d-e5fb-4ef5-af83-3662154a29fd","Type":"ContainerStarted","Data":"6400e660c81d4800612a5af86967e050276b27fb4faa22264edd614658f598ad"} Nov 02 17:40:12.728280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.728111 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" event={"ID":"2af0994d-e5fb-4ef5-af83-3662154a29fd","Type":"ContainerStarted","Data":"81d203ff2d6034a31c7047971f78a29ea6e98ecbed36d90f985de9be83156aa6"} Nov 02 17:40:12.732098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.731460 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2" event={"ID":"87d8bb05-337f-4eb5-b71f-f95f3b160ecb","Type":"ContainerStarted","Data":"16f4a5b3de427112aa9f92689582c4ff0d59b3d4ecf45fb0a02525fd0a6f42e4"} Nov 02 17:40:12.735223 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.735134 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zg6hp" event={"ID":"5350eca9-94c7-4dc9-a571-ec16e055d94d","Type":"ContainerStarted","Data":"eae54d49c0af3ede9209eaf5d651c3f317ca906674329744369c153c547481cc"} Nov 02 17:40:12.735223 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.735189 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zg6hp" event={"ID":"5350eca9-94c7-4dc9-a571-ec16e055d94d","Type":"ContainerStarted","Data":"e0f46a483ffbbaa8a2f31ee07fc3ad58b9d7349c06eeba8e2f305b80c965d6be"} Nov 02 17:40:12.738263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.738234 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d94c8b5d9-8gspc" event={"ID":"e324050f-2d8c-49fb-9763-0c1df35aab9a","Type":"ContainerStarted","Data":"3ccdf295ded318746969984f48c311ebb3fb6780ba655cdfa3854237c4dabf2c"} Nov 02 17:40:12.740185 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.740159 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tz2kg" event={"ID":"f27a1622-72d7-411f-baba-028b6c2f94ee","Type":"ContainerStarted","Data":"913ebe41d7745f22fc64613510004b8b18c0bfe7704cebe8b6306c37218dbb08"} Nov 02 17:40:12.746249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.746222 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7r6cv" event={"ID":"486c0825-8a27-4863-bbb7-27e11440fb83","Type":"ContainerStarted","Data":"44e4dcb1b88694ed72b1830dfd3b6e3aaf77d7a8c3cf3e691d44d061a81f2c2e"} Nov 02 17:40:12.750299 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.750243 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" podStartSLOduration=1.750227127 podStartE2EDuration="1.750227127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:12.749868878 +0000 UTC m=+10.192951530" watchObservedRunningTime="2026-11-02 17:40:12.750227127 +0000 UTC m=+10.193309765" Nov 02 17:40:12.991478 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:12.991435 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:13.084315 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:13.084279 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:13.754796 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:13.754199 4310 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:40:13.754796 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:13.754709 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zg6hp" event={"ID":"5350eca9-94c7-4dc9-a571-ec16e055d94d","Type":"ContainerStarted","Data":"176d900b13be48ad4221ca15e7bf7a5e51a48a53130c565ae1fcf90f5dbb82e7"} Nov 02 17:40:14.763499 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:14.763462 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2" event={"ID":"87d8bb05-337f-4eb5-b71f-f95f3b160ecb","Type":"ContainerStarted","Data":"dfbd0377d8645e320dcd8e53fe181a31c7a46000bea6bf17233047720b60e033"} Nov 02 17:40:15.766936 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:15.763427 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:15.771597 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:15.771565 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tz2kg" event={"ID":"f27a1622-72d7-411f-baba-028b6c2f94ee","Type":"ContainerStarted","Data":"95248837298b3f09eff33b4abd26215b966f422aff59c4890eedf1a2fe7b8b0f"} Nov 02 17:40:15.771736 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:15.771598 4310 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:40:15.771736 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:15.771634 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tz2kg" event={"ID":"f27a1622-72d7-411f-baba-028b6c2f94ee","Type":"ContainerStarted","Data":"6908b491c3e21dd1bf97851dbab5c910e6dfd9fead56302d7399e9edbb4269f0"} Nov 02 17:40:15.790899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:15.790852 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-tz2kg" podStartSLOduration=1.921121793 podStartE2EDuration="4.790837895s" totalImagesPullingTime="2.869716102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:15.78974044 +0000 UTC m=+13.232823086" watchObservedRunningTime="2026-11-02 17:40:15.790837895 +0000 UTC m=+13.233920531" Nov 02 17:40:15.792112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:15.791109 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2" podStartSLOduration=3.146303247 podStartE2EDuration="4.791099756s" totalImagesPullingTime="1.644796509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:14.783877722 +0000 UTC m=+12.226960359" watchObservedRunningTime="2026-11-02 17:40:15.791099756 +0000 UTC m=+13.234182386" Nov 02 17:40:15.869104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:15.869044 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:16.064762 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:16.064685 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:16.780530 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:16.779778 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d94c8b5d9-8gspc" event={"ID":"e324050f-2d8c-49fb-9763-0c1df35aab9a","Type":"ContainerStarted","Data":"81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470"} Nov 02 17:40:16.782216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:16.782189 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7r6cv" event={"ID":"486c0825-8a27-4863-bbb7-27e11440fb83","Type":"ContainerStarted","Data":"27f2b44af3e648739e7ff80e933d9f6ca52c93b70c6e6c9842e30066c919e540"} Nov 02 17:40:16.786001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:16.785970 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zg6hp" event={"ID":"5350eca9-94c7-4dc9-a571-ec16e055d94d","Type":"ContainerStarted","Data":"e86565fbcba0eeaa8e7ebd27ff25e18ed1638fa1b1236e90e82fc514a3e6af80"} Nov 02 17:40:16.786154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:16.786024 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:16.801863 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:16.801801 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7d94c8b5d9-8gspc" podStartSLOduration=1.321777976 podStartE2EDuration="5.801781858s" totalImagesPullingTime="4.480003882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:16.798987129 +0000 UTC m=+14.242069768" watchObservedRunningTime="2026-11-02 17:40:16.801781858 +0000 UTC m=+14.244864491" Nov 02 17:40:16.820491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:16.820434 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zg6hp" podStartSLOduration=2.442164537 podStartE2EDuration="5.82041919s" totalImagesPullingTime="3.378254653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:40:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:16.820273138 +0000 UTC m=+14.263355775" watchObservedRunningTime="2026-11-02 17:40:16.82041919 +0000 UTC m=+14.263501825" Nov 02 17:40:16.838418 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:16.838337 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7r6cv" podStartSLOduration=1.639465309 podStartE2EDuration="5.838320897s" totalImagesPullingTime="4.198855588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:16.83776273 +0000 UTC m=+14.280845367" watchObservedRunningTime="2026-11-02 17:40:16.838320897 +0000 UTC m=+14.281403525" Nov 02 17:40:17.857471 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:17.857418 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d94c8b5d9-8gspc_e324050f-2d8c-49fb-9763-0c1df35aab9a/console/0.log" Nov 02 17:40:18.174990 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:18.174868 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:18.218029 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:18.217997 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:18.272063 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:18.272024 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:18.277988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:18.277959 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:18.282163 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:18.282136 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:18.657011 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:18.656981 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tz2kg_f27a1622-72d7-411f-baba-028b6c2f94ee/dns/0.log" Nov 02 17:40:18.857103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:18.857063 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tz2kg_f27a1622-72d7-411f-baba-028b6c2f94ee/kube-rbac-proxy/0.log" Nov 02 17:40:19.457423 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:19.457393 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-59bfz_4307ecfe-be36-4891-9c9d-4c649e9aca70/dns-node-resolver/0.log" Nov 02 17:40:19.795408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:19.795379 4310 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:40:20.261271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:20.261238 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c84fd649c-lxh8c_2af0994d-e5fb-4ef5-af83-3662154a29fd/registry/0.log" Nov 02 17:40:20.856186 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:20.856162 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2crqk_a3b7e801-25bc-4f6e-be57-f3131d2ce39c/node-ca/0.log" Nov 02 17:40:21.104771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:21.104735 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:21.104929 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:21.104871 4310 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:40:21.109576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:21.109508 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:21.162248 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:21.162169 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:21.263221 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:21.263183 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:21.456691 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:21.456622 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7r6cv_486c0825-8a27-4863-bbb7-27e11440fb83/serve-healthcheck-canary/0.log" Nov 02 17:40:21.623868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:21.623837 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2" Nov 02 17:40:21.624034 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:21.623975 4310 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:40:21.628147 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:21.628126 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2" Nov 02 17:40:22.646777 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:22.646741 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:22.743132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:22.743099 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:22.857740 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:22.857701 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:23.365168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:23.365130 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:24.262737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:24.262696 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d94c8b5d9-8gspc"] Nov 02 17:40:24.372368 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:24.372333 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:24.473184 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:24.473149 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:25.339998 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:25.339965 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:26.791369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:26.791341 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tz2kg" Nov 02 17:40:26.802546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:26.802521 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:26.975563 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:26.975535 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8hhd" Nov 02 17:40:27.016050 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.016013 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:27.054863 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.054777 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-85vm8"] Nov 02 17:40:27.059512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.059478 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.088720 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.088691 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 17:40:27.088883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.088864 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 17:40:27.088966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.088919 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-84l5b\"" Nov 02 17:40:27.128833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.128796 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-wtmp\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.128988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.128865 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ac8dbad-7669-486e-9580-580b89796ba0-sys\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.129068 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.129049 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ac8dbad-7669-486e-9580-580b89796ba0-metrics-client-ca\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.129156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.129119 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.129212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.129154 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-tls\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.129212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.129194 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ac8dbad-7669-486e-9580-580b89796ba0-root\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.129315 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.129247 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2jj8\" (UniqueName: \"kubernetes.io/projected/6ac8dbad-7669-486e-9580-580b89796ba0-kube-api-access-l2jj8\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.129315 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.129292 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-accelerators-collector-config\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.129407 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.129326 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-textfile\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.143519 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.143483 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:27.165629 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.165605 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:40:27.166200 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.166144 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:40:27.166295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.166242 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 17:40:27.166295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.166277 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 17:40:27.229746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.229711 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-textfile\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.229746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.229751 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-wtmp\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.229958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.229796 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ac8dbad-7669-486e-9580-580b89796ba0-sys\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.229958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.229826 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ac8dbad-7669-486e-9580-580b89796ba0-metrics-client-ca\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.229958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.229847 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.229958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.229869 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-tls\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.229958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.229888 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ac8dbad-7669-486e-9580-580b89796ba0-root\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.229958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.229916 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2jj8\" (UniqueName: \"kubernetes.io/projected/6ac8dbad-7669-486e-9580-580b89796ba0-kube-api-access-l2jj8\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.229958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.229936 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-accelerators-collector-config\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.230195 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.230060 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-textfile\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.230398 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.230375 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-accelerators-collector-config\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.230398 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.230384 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ac8dbad-7669-486e-9580-580b89796ba0-sys\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.230567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.230502 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ac8dbad-7669-486e-9580-580b89796ba0-root\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.230567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.230522 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-wtmp\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.230804 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.230788 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ac8dbad-7669-486e-9580-580b89796ba0-metrics-client-ca\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.232607 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.232584 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.232754 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.232731 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ac8dbad-7669-486e-9580-580b89796ba0-node-exporter-tls\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.243091 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.243050 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2jj8\" (UniqueName: \"kubernetes.io/projected/6ac8dbad-7669-486e-9580-580b89796ba0-kube-api-access-l2jj8\") pod \"node-exporter-85vm8\" (UID: \"6ac8dbad-7669-486e-9580-580b89796ba0\") " pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.322615 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.322532 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:27.377265 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.377234 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-85vm8" Nov 02 17:40:27.390489 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:27.390459 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac8dbad_7669_486e_9580_580b89796ba0.slice/crio-55fee0dd6c9c26bdbebbda4bc44e12aac3f7ff15342b0babe15cbcdd95aeebfe WatchSource:0}: Error finding container 55fee0dd6c9c26bdbebbda4bc44e12aac3f7ff15342b0babe15cbcdd95aeebfe: Status 404 returned error can't find the container with id 55fee0dd6c9c26bdbebbda4bc44e12aac3f7ff15342b0babe15cbcdd95aeebfe Nov 02 17:40:27.424939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.424904 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:27.528788 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.528752 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:27.825282 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:27.825244 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-85vm8" event={"ID":"6ac8dbad-7669-486e-9580-580b89796ba0","Type":"ContainerStarted","Data":"55fee0dd6c9c26bdbebbda4bc44e12aac3f7ff15342b0babe15cbcdd95aeebfe"} Nov 02 17:40:28.090978 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:28.090891 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:28.223092 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:28.223047 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c84fd649c-lxh8c" Nov 02 17:40:28.250540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:28.250476 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:28.388187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:28.388110 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:28.588432 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:28.588394 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:29.153281 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.153200 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6c47fdcc77-94c69"] Nov 02 17:40:29.170503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.170472 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c47fdcc77-94c69"] Nov 02 17:40:29.170679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.170613 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.174775 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.174749 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 17:40:29.178216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.178195 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ft9kv5i2rl1dk\"" Nov 02 17:40:29.249404 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.249360 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.249404 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.249409 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-tls\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.249606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.249507 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.249606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.249534 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7tsr\" (UniqueName: \"kubernetes.io/projected/54365f2d-6ca2-4f59-8c06-c5406f9857e3-kube-api-access-s7tsr\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.249673 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.249625 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54365f2d-6ca2-4f59-8c06-c5406f9857e3-tmp\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.249673 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.249646 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-grpc-tls\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.249733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.249669 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.249733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.249710 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.249794 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.249744 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54365f2d-6ca2-4f59-8c06-c5406f9857e3-metrics-client-ca\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.275579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.275543 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 17:40:29.275767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.275637 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 17:40:29.275767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.275679 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 17:40:29.275767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.275755 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 17:40:29.276033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.276021 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tw6z2\"" Nov 02 17:40:29.350145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.350112 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.350145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.350147 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7tsr\" (UniqueName: \"kubernetes.io/projected/54365f2d-6ca2-4f59-8c06-c5406f9857e3-kube-api-access-s7tsr\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.350350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.350171 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54365f2d-6ca2-4f59-8c06-c5406f9857e3-tmp\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.350350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.350282 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-grpc-tls\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.350350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.350324 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.350505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.350351 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.350505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.350377 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54365f2d-6ca2-4f59-8c06-c5406f9857e3-metrics-client-ca\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.350608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.350520 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.350608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.350551 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54365f2d-6ca2-4f59-8c06-c5406f9857e3-tmp\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.350608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.350587 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-tls\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.351289 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.351263 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54365f2d-6ca2-4f59-8c06-c5406f9857e3-metrics-client-ca\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.353327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.353281 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.353439 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.353398 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.353505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.353457 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.353545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.353507 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-tls\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.353545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.353512 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-grpc-tls\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.353801 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.353784 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/54365f2d-6ca2-4f59-8c06-c5406f9857e3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.358552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.358479 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7tsr\" (UniqueName: \"kubernetes.io/projected/54365f2d-6ca2-4f59-8c06-c5406f9857e3-kube-api-access-s7tsr\") pod \"thanos-querier-6c47fdcc77-94c69\" (UID: \"54365f2d-6ca2-4f59-8c06-c5406f9857e3\") " pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.572224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.572192 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:29.835034 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.834939 4310 generic.go:356] "Generic (PLEG): container finished" podID="6ac8dbad-7669-486e-9580-580b89796ba0" containerID="62956253658d1d0fbfaf90937f67ff94dbacb5b0816a74deaafab8161c271d26" exitCode=0 Nov 02 17:40:29.835034 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.835022 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-85vm8" event={"ID":"6ac8dbad-7669-486e-9580-580b89796ba0","Type":"ContainerDied","Data":"62956253658d1d0fbfaf90937f67ff94dbacb5b0816a74deaafab8161c271d26"} Nov 02 17:40:29.837348 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.837319 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-57wqv" event={"ID":"a34b1f69-0317-462a-981b-51ae91f9e8b5","Type":"ContainerStarted","Data":"4a5b115b06fd35e83335cac7e98da7b307f346a8ac6e75a430dedbcf1ee8345e"} Nov 02 17:40:29.837749 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.837722 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-57wqv" Nov 02 17:40:29.852424 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.852390 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c47fdcc77-94c69"] Nov 02 17:40:29.854779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.854348 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-57wqv" Nov 02 17:40:29.856207 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:29.856178 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54365f2d_6ca2_4f59_8c06_c5406f9857e3.slice/crio-6b5077885e86642728814efcc5f0dce84c2d707de0631dc0cceef1c4b48768b5 WatchSource:0}: Error finding container 6b5077885e86642728814efcc5f0dce84c2d707de0631dc0cceef1c4b48768b5: Status 404 returned error can't find the container with id 6b5077885e86642728814efcc5f0dce84c2d707de0631dc0cceef1c4b48768b5 Nov 02 17:40:29.879575 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:29.879514 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-57wqv" podStartSLOduration=1.53647635 podStartE2EDuration="18.879495104s" totalImagesPullingTime="17.343018754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:29.878502039 +0000 UTC m=+27.321584677" watchObservedRunningTime="2026-11-02 17:40:29.879495104 +0000 UTC m=+27.322577740" Nov 02 17:40:30.032715 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:30.032646 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:30.130686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:30.130601 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:30.234165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:30.234130 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:30.331287 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:30.331208 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:30.844172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:30.844135 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" event={"ID":"54365f2d-6ca2-4f59-8c06-c5406f9857e3","Type":"ContainerStarted","Data":"6b5077885e86642728814efcc5f0dce84c2d707de0631dc0cceef1c4b48768b5"} Nov 02 17:40:30.848947 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:30.848853 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-85vm8" event={"ID":"6ac8dbad-7669-486e-9580-580b89796ba0","Type":"ContainerStarted","Data":"cd7b8367634bb8172cae9260ef9cc25c035d9a2856a5718a3290807ddeee6889"} Nov 02 17:40:30.848947 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:30.848908 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-85vm8" event={"ID":"6ac8dbad-7669-486e-9580-580b89796ba0","Type":"ContainerStarted","Data":"d403476123c8a2c4a9df2b77c701065ea476830429b08424c7bfda933f1e0fd3"} Nov 02 17:40:31.447817 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.447738 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-85vm8" podStartSLOduration=2.9567908259999998 podStartE2EDuration="4.447715691s" totalImagesPullingTime="1.397256025s" totalInitContainerRuntime="93.66884ms" isStatefulPod=true podCreationTimestamp="2026-11-02 17:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:30.872113213 +0000 UTC m=+28.315195851" watchObservedRunningTime="2026-11-02 17:40:31.447715691 +0000 UTC m=+28.890798328" Nov 02 17:40:31.448555 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.448527 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bfccc956c-622tf"] Nov 02 17:40:31.484211 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.482047 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bfccc956c-622tf"] Nov 02 17:40:31.484211 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.482141 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.486463 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.486432 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 17:40:31.487280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.487259 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 17:40:31.487946 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.487608 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 17:40:31.487946 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.487759 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ekg53nf8f8pug\"" Nov 02 17:40:31.487946 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.487844 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wnnbb\"" Nov 02 17:40:31.487946 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.487914 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 17:40:31.569279 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.569217 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dfe31f7-f071-4aac-aa92-ae6509305835-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.569279 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.569279 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6dfe31f7-f071-4aac-aa92-ae6509305835-secret-metrics-server-tls\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.569540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.569310 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6dfe31f7-f071-4aac-aa92-ae6509305835-client-ca-bundle\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.569540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.569346 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmr9m\" (UniqueName: \"kubernetes.io/projected/6dfe31f7-f071-4aac-aa92-ae6509305835-kube-api-access-pmr9m\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.569540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.569379 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6dfe31f7-f071-4aac-aa92-ae6509305835-secret-metrics-server-client-certs\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.569540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.569411 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6dfe31f7-f071-4aac-aa92-ae6509305835-metrics-server-audit-profiles\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.569540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.569465 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6dfe31f7-f071-4aac-aa92-ae6509305835-audit-log\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.670803 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.670746 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr9m\" (UniqueName: \"kubernetes.io/projected/6dfe31f7-f071-4aac-aa92-ae6509305835-kube-api-access-pmr9m\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.671004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.670807 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6dfe31f7-f071-4aac-aa92-ae6509305835-secret-metrics-server-client-certs\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.671004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.670849 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6dfe31f7-f071-4aac-aa92-ae6509305835-metrics-server-audit-profiles\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.671004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.670889 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6dfe31f7-f071-4aac-aa92-ae6509305835-audit-log\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.671004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.670929 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dfe31f7-f071-4aac-aa92-ae6509305835-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.671004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.670977 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6dfe31f7-f071-4aac-aa92-ae6509305835-secret-metrics-server-tls\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.671303 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.671191 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6dfe31f7-f071-4aac-aa92-ae6509305835-client-ca-bundle\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.673315 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.671933 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6dfe31f7-f071-4aac-aa92-ae6509305835-audit-log\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.673315 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.672420 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dfe31f7-f071-4aac-aa92-ae6509305835-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.673315 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.673275 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6dfe31f7-f071-4aac-aa92-ae6509305835-metrics-server-audit-profiles\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.675206 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.674559 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6dfe31f7-f071-4aac-aa92-ae6509305835-secret-metrics-server-tls\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.675206 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.674567 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6dfe31f7-f071-4aac-aa92-ae6509305835-client-ca-bundle\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.680147 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.679378 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmr9m\" (UniqueName: \"kubernetes.io/projected/6dfe31f7-f071-4aac-aa92-ae6509305835-kube-api-access-pmr9m\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.683061 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.683034 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6dfe31f7-f071-4aac-aa92-ae6509305835-secret-metrics-server-client-certs\") pod \"metrics-server-bfccc956c-622tf\" (UID: \"6dfe31f7-f071-4aac-aa92-ae6509305835\") " pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.816588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.816551 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:31.850813 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.850766 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:31.974045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:31.974008 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:32.072529 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:32.072445 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:32.139696 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:32.139658 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bfccc956c-622tf"] Nov 02 17:40:32.152954 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:32.152921 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfe31f7_f071_4aac_aa92_ae6509305835.slice/crio-50f7cb53fed701de5fb68ac6e32c60300e3c411c7d1c1c54457214793068fadb WatchSource:0}: Error finding container 50f7cb53fed701de5fb68ac6e32c60300e3c411c7d1c1c54457214793068fadb: Status 404 returned error can't find the container with id 50f7cb53fed701de5fb68ac6e32c60300e3c411c7d1c1c54457214793068fadb Nov 02 17:40:32.858518 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:32.858434 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bfccc956c-622tf" event={"ID":"6dfe31f7-f071-4aac-aa92-ae6509305835","Type":"ContainerStarted","Data":"50f7cb53fed701de5fb68ac6e32c60300e3c411c7d1c1c54457214793068fadb"} Nov 02 17:40:32.861205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:32.861177 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" event={"ID":"54365f2d-6ca2-4f59-8c06-c5406f9857e3","Type":"ContainerStarted","Data":"a62da1e5844c81562e0af138c489f0b2507d388343314ff180bfba93d62a6a6b"} Nov 02 17:40:32.861318 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:32.861251 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" event={"ID":"54365f2d-6ca2-4f59-8c06-c5406f9857e3","Type":"ContainerStarted","Data":"5c6b1b18ec2fe1790c72315eda54a497fa84dd60a11b83b6bff014b2f0d789e4"} Nov 02 17:40:33.416337 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:33.416298 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:33.529997 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:40:33.529766 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:40:33.616572 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:33.616536 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:33.717011 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:33.716973 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:33.869341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:33.869240 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" event={"ID":"54365f2d-6ca2-4f59-8c06-c5406f9857e3","Type":"ContainerStarted","Data":"36798307e72cdc0e806bc612c37a1fbb417ffb9edc2ad9dbde6b6c83aa8f050f"} Nov 02 17:40:34.876281 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:34.875782 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bfccc956c-622tf" event={"ID":"6dfe31f7-f071-4aac-aa92-ae6509305835","Type":"ContainerStarted","Data":"87939821ed30bf1539c1c2c21d10c90ce10280dc3458de6544c5f1afcf67d24d"} Nov 02 17:40:34.879531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:34.879502 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" event={"ID":"54365f2d-6ca2-4f59-8c06-c5406f9857e3","Type":"ContainerStarted","Data":"809a2ce72007de766f4ddb0b40f618ef6eb43ce6eb18d1d29ce3018da1b9778f"} Nov 02 17:40:34.879623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:34.879575 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" event={"ID":"54365f2d-6ca2-4f59-8c06-c5406f9857e3","Type":"ContainerStarted","Data":"9bd56d612392a1b4c927bf619d9ac23051b5e424c33c754e519c59a727a69f58"} Nov 02 17:40:35.352123 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.351831 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bfccc956c-622tf" podStartSLOduration=1.961166093 podStartE2EDuration="4.351814079s" totalImagesPullingTime="2.390647986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:34.904992412 +0000 UTC m=+32.348075052" watchObservedRunningTime="2026-11-02 17:40:35.351814079 +0000 UTC m=+32.794896716" Nov 02 17:40:35.353100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.353013 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-848c84fbb4-w8474"] Nov 02 17:40:35.379410 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.379235 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848c84fbb4-w8474"] Nov 02 17:40:35.379410 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.379393 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.386970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.386885 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:35.392792 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.392760 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 17:40:35.491684 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.491640 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:35.504651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.504609 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-config\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.504828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.504667 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-serving-cert\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.504828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.504760 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-tmp\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.504950 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.504876 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-trusted-ca-bundle\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.504950 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.504897 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttjkz\" (UniqueName: \"kubernetes.io/projected/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-kube-api-access-ttjkz\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.504950 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.504925 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-oauth-config\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.505150 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.504947 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-service-ca\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.505150 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.505109 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-oauth-serving-cert\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.609546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.608365 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-serving-cert\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.609546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.608402 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-tmp\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.609546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.608439 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-trusted-ca-bundle\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.609546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.608455 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttjkz\" (UniqueName: \"kubernetes.io/projected/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-kube-api-access-ttjkz\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.609546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.608476 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-oauth-config\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.609546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.608494 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-service-ca\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.609546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.608517 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-oauth-serving-cert\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.609546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.608541 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-config\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.609546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.609090 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-config\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.610908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.610883 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-tmp\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.612878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.611892 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-service-ca\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.612994 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.612400 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-oauth-serving-cert\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.612994 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.612808 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-trusted-ca-bundle\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.617829 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.617807 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-serving-cert\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.620530 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.620502 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-oauth-config\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.620634 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.620580 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttjkz\" (UniqueName: \"kubernetes.io/projected/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-kube-api-access-ttjkz\") pod \"console-848c84fbb4-w8474\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.714630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.714550 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:35.920062 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.919965 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" event={"ID":"54365f2d-6ca2-4f59-8c06-c5406f9857e3","Type":"ContainerStarted","Data":"e97f5f613ff1935d37b7ca00d8bf5c224f8a1e5325fd51f3a97119c61a4148ef"} Nov 02 17:40:35.949872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:35.949820 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" podStartSLOduration=2.623494395 podStartE2EDuration="6.949805623s" totalImagesPullingTime="4.326311228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:35.947950838 +0000 UTC m=+33.391033465" watchObservedRunningTime="2026-11-02 17:40:35.949805623 +0000 UTC m=+33.392888259" Nov 02 17:40:36.079310 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:36.079061 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848c84fbb4-w8474"] Nov 02 17:40:36.106054 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:36.106017 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77d84e5_4e2b_49b6_ac0d_97ec9d504e62.slice/crio-72df021aee8dc9c004496b8281465ccb31319e281a90b16183962c4cc162500f WatchSource:0}: Error finding container 72df021aee8dc9c004496b8281465ccb31319e281a90b16183962c4cc162500f: Status 404 returned error can't find the container with id 72df021aee8dc9c004496b8281465ccb31319e281a90b16183962c4cc162500f Nov 02 17:40:36.242832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:36.242795 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:36.341274 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:36.341241 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:36.925846 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:36.925752 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848c84fbb4-w8474" event={"ID":"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62","Type":"ContainerStarted","Data":"f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715"} Nov 02 17:40:36.925846 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:36.925810 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848c84fbb4-w8474" event={"ID":"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62","Type":"ContainerStarted","Data":"72df021aee8dc9c004496b8281465ccb31319e281a90b16183962c4cc162500f"} Nov 02 17:40:36.926430 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:36.926118 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:36.945332 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:36.945268 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-848c84fbb4-w8474" podStartSLOduration=1.945251092 podStartE2EDuration="1.945251092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:36.944025807 +0000 UTC m=+34.387108477" watchObservedRunningTime="2026-11-02 17:40:36.945251092 +0000 UTC m=+34.388333729" Nov 02 17:40:37.928853 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:37.928812 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x4786" Nov 02 17:40:37.936587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:37.936565 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" Nov 02 17:40:41.153515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.153481 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848c84fbb4-w8474"] Nov 02 17:40:41.183873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.183836 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b76874c59-2mf6g"] Nov 02 17:40:41.209674 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.209645 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b76874c59-2mf6g"] Nov 02 17:40:41.209835 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.209740 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.370561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.370521 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-serving-cert\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.370807 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.370567 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-service-ca\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.370807 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.370593 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-tmp\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.370807 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.370707 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-config\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.370807 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.370760 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-oauth-config\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.370966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.370828 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6rj8\" (UniqueName: \"kubernetes.io/projected/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-kube-api-access-b6rj8\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.370966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.370863 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-trusted-ca-bundle\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.370966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.370887 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-oauth-serving-cert\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.471536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.471500 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-oauth-config\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.471680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.471561 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6rj8\" (UniqueName: \"kubernetes.io/projected/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-kube-api-access-b6rj8\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.471680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.471584 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-trusted-ca-bundle\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.471680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.471601 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-oauth-serving-cert\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.471680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.471654 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-serving-cert\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.471869 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.471779 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-service-ca\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.471869 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.471820 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-tmp\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.471869 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.471852 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-config\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.472356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.472327 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-tmp\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.472520 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.472493 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-oauth-serving-cert\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.472584 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.472557 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-service-ca\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.472643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.472622 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-config\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.472704 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.472689 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-trusted-ca-bundle\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.473928 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.473901 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-oauth-config\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.474035 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.474019 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-serving-cert\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.488263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.488240 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6rj8\" (UniqueName: \"kubernetes.io/projected/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-kube-api-access-b6rj8\") pod \"console-5b76874c59-2mf6g\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.531831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.531793 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:41.831005 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.830928 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b76874c59-2mf6g"] Nov 02 17:40:41.834449 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:40:41.834420 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecc4c47_82f4_40fb_8f8d_762d3af66a0d.slice/crio-1a083402ff74c3e4a06ce8bdde932487e15f212456bbe5650e6237e062aa0597 WatchSource:0}: Error finding container 1a083402ff74c3e4a06ce8bdde932487e15f212456bbe5650e6237e062aa0597: Status 404 returned error can't find the container with id 1a083402ff74c3e4a06ce8bdde932487e15f212456bbe5650e6237e062aa0597 Nov 02 17:40:41.945340 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.945301 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b76874c59-2mf6g" event={"ID":"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d","Type":"ContainerStarted","Data":"a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f"} Nov 02 17:40:41.945534 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.945367 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b76874c59-2mf6g" event={"ID":"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d","Type":"ContainerStarted","Data":"1a083402ff74c3e4a06ce8bdde932487e15f212456bbe5650e6237e062aa0597"} Nov 02 17:40:41.969442 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.969389 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b76874c59-2mf6g" podStartSLOduration=0.969372336 podStartE2EDuration="969.372336ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:41.96796038 +0000 UTC m=+39.411043015" watchObservedRunningTime="2026-11-02 17:40:41.969372336 +0000 UTC m=+39.412454972" Nov 02 17:40:41.971708 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:41.971681 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:42.389454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:42.389425 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:42.496398 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:42.496357 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:42.649500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:42.649425 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:42.784973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:42.784942 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:42.986038 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:42.986004 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:43.087559 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:43.087529 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:44.205398 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:44.205360 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:44.303579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:44.303546 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:45.291413 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:45.291381 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:45.413798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:45.413764 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:45.714794 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:45.714769 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:40:49.284795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.284729 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7d94c8b5d9-8gspc" podUID="e324050f-2d8c-49fb-9763-0c1df35aab9a" containerName="console" containerID="cri-o://81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470" gracePeriod=15 Nov 02 17:40:49.561643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.561567 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:49.639358 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.639316 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-config\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-config\") pod \"e324050f-2d8c-49fb-9763-0c1df35aab9a\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " Nov 02 17:40:49.639542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.639369 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-serving-cert\") pod \"e324050f-2d8c-49fb-9763-0c1df35aab9a\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " Nov 02 17:40:49.639542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.639392 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e324050f-2d8c-49fb-9763-0c1df35aab9a-kube-api-access-z48wn\" (UniqueName: \"kubernetes.io/projected/e324050f-2d8c-49fb-9763-0c1df35aab9a-kube-api-access-z48wn\") pod \"e324050f-2d8c-49fb-9763-0c1df35aab9a\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " Nov 02 17:40:49.639542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.639416 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-service-ca\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-service-ca\") pod \"e324050f-2d8c-49fb-9763-0c1df35aab9a\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " Nov 02 17:40:49.639696 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.639625 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e324050f-2d8c-49fb-9763-0c1df35aab9a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e324050f-2d8c-49fb-9763-0c1df35aab9a-tmp\") pod \"e324050f-2d8c-49fb-9763-0c1df35aab9a\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " Nov 02 17:40:49.639843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.639814 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-config" pod "e324050f-2d8c-49fb-9763-0c1df35aab9a" (UID: "e324050f-2d8c-49fb-9763-0c1df35aab9a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:40:49.639970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.639853 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-service-ca" pod "e324050f-2d8c-49fb-9763-0c1df35aab9a" (UID: "e324050f-2d8c-49fb-9763-0c1df35aab9a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:40:49.639970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.639898 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e324050f-2d8c-49fb-9763-0c1df35aab9a-tmp" pod "e324050f-2d8c-49fb-9763-0c1df35aab9a" (UID: "e324050f-2d8c-49fb-9763-0c1df35aab9a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:40:49.641832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.641808 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-serving-cert" pod "e324050f-2d8c-49fb-9763-0c1df35aab9a" (UID: "e324050f-2d8c-49fb-9763-0c1df35aab9a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:40:49.641924 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.641886 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e324050f-2d8c-49fb-9763-0c1df35aab9a-kube-api-access-z48wn" pod "e324050f-2d8c-49fb-9763-0c1df35aab9a" (UID: "e324050f-2d8c-49fb-9763-0c1df35aab9a"). InnerVolumeSpecName "kube-api-access-z48wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:40:49.740157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.740109 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-oauth-config\") pod \"e324050f-2d8c-49fb-9763-0c1df35aab9a\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " Nov 02 17:40:49.740157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.740154 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-oauth-serving-cert\") pod \"e324050f-2d8c-49fb-9763-0c1df35aab9a\" (UID: \"e324050f-2d8c-49fb-9763-0c1df35aab9a\") " Nov 02 17:40:49.740367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.740267 4310 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-config\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:40:49.740367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.740280 4310 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-serving-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:40:49.740367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.740289 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z48wn\" (UniqueName: \"kubernetes.io/projected/e324050f-2d8c-49fb-9763-0c1df35aab9a-kube-api-access-z48wn\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:40:49.740367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.740298 4310 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-service-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:40:49.740367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.740306 4310 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e324050f-2d8c-49fb-9763-0c1df35aab9a-tmp\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:40:49.740570 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.740542 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-oauth-serving-cert" pod "e324050f-2d8c-49fb-9763-0c1df35aab9a" (UID: "e324050f-2d8c-49fb-9763-0c1df35aab9a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:40:49.742286 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.742264 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-oauth-config" pod "e324050f-2d8c-49fb-9763-0c1df35aab9a" (UID: "e324050f-2d8c-49fb-9763-0c1df35aab9a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:40:49.841240 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.841172 4310 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e324050f-2d8c-49fb-9763-0c1df35aab9a-console-oauth-config\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:40:49.841240 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.841197 4310 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e324050f-2d8c-49fb-9763-0c1df35aab9a-oauth-serving-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:40:49.974072 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.974034 4310 generic.go:356] "Generic (PLEG): container finished" podID="e324050f-2d8c-49fb-9763-0c1df35aab9a" containerID="81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470" exitCode=0 Nov 02 17:40:49.974269 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.974100 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d94c8b5d9-8gspc" Nov 02 17:40:49.974269 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.974118 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d94c8b5d9-8gspc" event={"ID":"e324050f-2d8c-49fb-9763-0c1df35aab9a","Type":"ContainerDied","Data":"81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470"} Nov 02 17:40:49.974269 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.974167 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d94c8b5d9-8gspc" event={"ID":"e324050f-2d8c-49fb-9763-0c1df35aab9a","Type":"ContainerDied","Data":"3ccdf295ded318746969984f48c311ebb3fb6780ba655cdfa3854237c4dabf2c"} Nov 02 17:40:49.974269 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.974186 4310 scope.go:122] "RemoveContainer" containerID="81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470" Nov 02 17:40:49.982672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.982654 4310 scope.go:122] "RemoveContainer" containerID="81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470" Nov 02 17:40:49.982956 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:40:49.982934 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470\": container with ID starting with 81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470 not found: ID does not exist" containerID="81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470" Nov 02 17:40:49.983018 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.982967 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470"} err="failed to get container status \"81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470\": rpc error: code = NotFound desc = could not find container \"81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470\": container with ID starting with 81095d08b373c91bd68aaa45b65dcc3fca113b1e38bab56ad8c86e0b585fd470 not found: ID does not exist" Nov 02 17:40:49.995231 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.995203 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d94c8b5d9-8gspc"] Nov 02 17:40:49.999055 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:49.999029 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d94c8b5d9-8gspc"] Nov 02 17:40:50.670884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:50.670851 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e324050f-2d8c-49fb-9763-0c1df35aab9a" path="/var/lib/kubelet/pods/e324050f-2d8c-49fb-9763-0c1df35aab9a/volumes" Nov 02 17:40:51.532402 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:51.532366 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:51.532641 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:51.532429 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:51.537000 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:51.536975 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:51.817015 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:51.816932 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:51.817506 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:51.817022 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:40:51.988670 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:51.988646 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:40:57.621182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:57.621148 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:40:57.710477 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:40:57.710445 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:03.522331 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:41:03.522293 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:41:06.173000 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.172937 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-848c84fbb4-w8474" podUID="d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" containerName="console" containerID="cri-o://f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715" gracePeriod=15 Nov 02 17:41:06.249659 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.249626 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:06.362031 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.361988 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:06.447242 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.447175 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:41:06.461321 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.461283 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:06.463292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463275 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-tmp\") pod \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " Nov 02 17:41:06.463348 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463309 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-oauth-serving-cert\") pod \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " Nov 02 17:41:06.463348 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463341 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-service-ca\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-service-ca\") pod \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " Nov 02 17:41:06.463427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463363 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-config\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-config\") pod \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " Nov 02 17:41:06.463427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463384 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-trusted-ca-bundle\") pod \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " Nov 02 17:41:06.463427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463405 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-serving-cert\") pod \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " Nov 02 17:41:06.463561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463476 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-kube-api-access-ttjkz\" (UniqueName: \"kubernetes.io/projected/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-kube-api-access-ttjkz\") pod \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " Nov 02 17:41:06.463561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463506 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-oauth-config\") pod \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\" (UID: \"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62\") " Nov 02 17:41:06.463658 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463552 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-tmp" pod "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" (UID: "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:06.463717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463668 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-oauth-serving-cert" pod "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" (UID: "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:06.463775 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463724 4310 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-tmp\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:06.463827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463777 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-service-ca" pod "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" (UID: "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:06.463827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463792 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-config" pod "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" (UID: "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:06.463921 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.463861 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-trusted-ca-bundle" pod "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" (UID: "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:06.465681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.465649 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-kube-api-access-ttjkz" pod "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" (UID: "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62"). InnerVolumeSpecName "kube-api-access-ttjkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:06.465816 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.465789 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-serving-cert" pod "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" (UID: "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:06.465816 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.465810 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-oauth-config" pod "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" (UID: "d77d84e5-4e2b-49b6-ac0d-97ec9d504e62"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:06.564094 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.564054 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttjkz\" (UniqueName: \"kubernetes.io/projected/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-kube-api-access-ttjkz\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:06.564094 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.564100 4310 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-oauth-config\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:06.564298 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.564115 4310 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-oauth-serving-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:06.564298 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.564124 4310 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-service-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:06.564298 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.564133 4310 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-config\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:06.564298 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.564141 4310 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-trusted-ca-bundle\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:06.564298 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:06.564150 4310 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62-console-serving-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:07.035565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:07.035534 4310 generic.go:356] "Generic (PLEG): container finished" podID="d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" containerID="f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715" exitCode=0 Nov 02 17:41:07.035741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:07.035574 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848c84fbb4-w8474" Nov 02 17:41:07.035741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:07.035612 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848c84fbb4-w8474" event={"ID":"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62","Type":"ContainerDied","Data":"f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715"} Nov 02 17:41:07.035741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:07.035660 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848c84fbb4-w8474" event={"ID":"d77d84e5-4e2b-49b6-ac0d-97ec9d504e62","Type":"ContainerDied","Data":"72df021aee8dc9c004496b8281465ccb31319e281a90b16183962c4cc162500f"} Nov 02 17:41:07.035741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:07.035680 4310 scope.go:122] "RemoveContainer" containerID="f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715" Nov 02 17:41:07.043587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:07.043568 4310 scope.go:122] "RemoveContainer" containerID="f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715" Nov 02 17:41:07.043850 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:41:07.043829 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715\": container with ID starting with f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715 not found: ID does not exist" containerID="f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715" Nov 02 17:41:07.043899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:07.043858 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715"} err="failed to get container status \"f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715\": rpc error: code = NotFound desc = could not find container \"f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715\": container with ID starting with f6a14bc4d256cf62d70af792adc5caf0095d416638d770bd1f9a0c69d72c0715 not found: ID does not exist" Nov 02 17:41:07.052141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:07.052119 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848c84fbb4-w8474"] Nov 02 17:41:07.055917 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:07.055900 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-848c84fbb4-w8474"] Nov 02 17:41:07.645556 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:07.645522 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:08.670925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:08.670897 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" path="/var/lib/kubelet/pods/d77d84e5-4e2b-49b6-ac0d-97ec9d504e62/volumes" Nov 02 17:41:11.821986 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:11.821956 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:41:11.825821 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:11.825801 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bfccc956c-622tf" Nov 02 17:41:12.572438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:12.572404 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:12.672922 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:12.672890 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:17.428752 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.428718 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b68b9d69f-jlqkc"] Nov 02 17:41:17.429160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.428999 4310 cpu_manager.go:414] "containerMap: removing container" podUID="e324050f-2d8c-49fb-9763-0c1df35aab9a" containerName="console" Nov 02 17:41:17.429160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.429010 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e324050f-2d8c-49fb-9763-0c1df35aab9a" containerName="console" Nov 02 17:41:17.429160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.429027 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" containerName="console" Nov 02 17:41:17.429160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.429032 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" containerName="console" Nov 02 17:41:17.429160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.429071 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77d84e5-4e2b-49b6-ac0d-97ec9d504e62" containerName="console" Nov 02 17:41:17.429160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.429096 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="e324050f-2d8c-49fb-9763-0c1df35aab9a" containerName="console" Nov 02 17:41:17.447781 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.447759 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b68b9d69f-jlqkc"] Nov 02 17:41:17.447919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.447848 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.522162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.522127 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:17.544579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.544547 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c27b5e-11b0-489c-b3cd-6eface16a294-tmp\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.544679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.544587 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdf2c\" (UniqueName: \"kubernetes.io/projected/24c27b5e-11b0-489c-b3cd-6eface16a294-kube-api-access-mdf2c\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.544679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.544618 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-console-config\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.544766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.544703 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24c27b5e-11b0-489c-b3cd-6eface16a294-console-oauth-config\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.544766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.544734 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-oauth-serving-cert\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.544766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.544759 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-service-ca\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.544872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.544841 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-trusted-ca-bundle\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.544955 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.544938 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24c27b5e-11b0-489c-b3cd-6eface16a294-console-serving-cert\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.621834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.621780 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:17.646071 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.646029 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24c27b5e-11b0-489c-b3cd-6eface16a294-console-oauth-config\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.646187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.646102 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-oauth-serving-cert\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.646187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.646129 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-service-ca\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.646187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.646148 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-trusted-ca-bundle\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.646356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.646193 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24c27b5e-11b0-489c-b3cd-6eface16a294-console-serving-cert\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.646356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.646229 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c27b5e-11b0-489c-b3cd-6eface16a294-tmp\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.646356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.646260 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdf2c\" (UniqueName: \"kubernetes.io/projected/24c27b5e-11b0-489c-b3cd-6eface16a294-kube-api-access-mdf2c\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.646356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.646299 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-console-config\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.646696 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.646660 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c27b5e-11b0-489c-b3cd-6eface16a294-tmp\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.647017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.646990 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-service-ca\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.647155 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.647106 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-console-config\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.647228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.647166 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-oauth-serving-cert\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.647285 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.647239 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24c27b5e-11b0-489c-b3cd-6eface16a294-trusted-ca-bundle\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.648562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.648538 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24c27b5e-11b0-489c-b3cd-6eface16a294-console-serving-cert\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.648662 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.648612 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24c27b5e-11b0-489c-b3cd-6eface16a294-console-oauth-config\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.656021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.655995 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdf2c\" (UniqueName: \"kubernetes.io/projected/24c27b5e-11b0-489c-b3cd-6eface16a294-kube-api-access-mdf2c\") pod \"console-5b68b9d69f-jlqkc\" (UID: \"24c27b5e-11b0-489c-b3cd-6eface16a294\") " pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.776306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.776269 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:17.866871 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.866837 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:17.966728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:17.966699 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:18.074216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:18.074147 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b68b9d69f-jlqkc"] Nov 02 17:41:18.078839 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:41:18.078807 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c27b5e_11b0_489c_b3cd_6eface16a294.slice/crio-84db5a75e5c0e1d576ee5629f1470cc5a66e386c916b7840e200f567859d6a4c WatchSource:0}: Error finding container 84db5a75e5c0e1d576ee5629f1470cc5a66e386c916b7840e200f567859d6a4c: Status 404 returned error can't find the container with id 84db5a75e5c0e1d576ee5629f1470cc5a66e386c916b7840e200f567859d6a4c Nov 02 17:41:19.078338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:19.078301 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b68b9d69f-jlqkc" event={"ID":"24c27b5e-11b0-489c-b3cd-6eface16a294","Type":"ContainerStarted","Data":"09692aa537be953cdcf2d87a9999b213eedef5d7721720433dcf6903d5b41c4d"} Nov 02 17:41:19.078725 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:19.078354 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b68b9d69f-jlqkc" event={"ID":"24c27b5e-11b0-489c-b3cd-6eface16a294","Type":"ContainerStarted","Data":"84db5a75e5c0e1d576ee5629f1470cc5a66e386c916b7840e200f567859d6a4c"} Nov 02 17:41:19.083286 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:19.083260 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:19.097563 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:19.097509 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b68b9d69f-jlqkc" podStartSLOduration=2.097490204 podStartE2EDuration="2.097490204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:19.097171644 +0000 UTC m=+76.540254283" watchObservedRunningTime="2026-11-02 17:41:19.097490204 +0000 UTC m=+76.540572840" Nov 02 17:41:19.184835 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:19.184800 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:27.637465 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:27.637424 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:27.739475 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:27.739405 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:27.776419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:27.776389 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:27.776565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:27.776450 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:27.781183 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:27.781153 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:27.837314 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:27.837276 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:28.112612 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:28.112585 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b68b9d69f-jlqkc" Nov 02 17:41:28.162553 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:28.162522 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b76874c59-2mf6g"] Nov 02 17:41:33.522780 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:41:33.522742 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:41:43.624999 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:43.624962 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:43.720603 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:43.720571 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:45.332128 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:45.332097 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:41:53.183437 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.183381 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5b76874c59-2mf6g" podUID="8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" containerName="console" containerID="cri-o://a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f" gracePeriod=15 Nov 02 17:41:53.417637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.417614 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:41:53.532582 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.532529 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-oauth-config\") pod \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " Nov 02 17:41:53.532582 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.532583 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-trusted-ca-bundle\") pod \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " Nov 02 17:41:53.532826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.532617 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-tmp\") pod \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " Nov 02 17:41:53.532826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.532634 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-oauth-serving-cert\") pod \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " Nov 02 17:41:53.532826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.532661 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-kube-api-access-b6rj8\" (UniqueName: \"kubernetes.io/projected/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-kube-api-access-b6rj8\") pod \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " Nov 02 17:41:53.532826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.532703 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-serving-cert\") pod \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " Nov 02 17:41:53.532826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.532744 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-config\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-config\") pod \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " Nov 02 17:41:53.532826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.532795 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-service-ca\") pod \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\" (UID: \"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d\") " Nov 02 17:41:53.533141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.532908 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-tmp" pod "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" (UID: "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:41:53.533141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.533060 4310 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-tmp\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:53.533249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.533177 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-oauth-serving-cert" pod "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" (UID: "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:53.533249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.533215 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-trusted-ca-bundle" pod "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" (UID: "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:53.533249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.533226 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-service-ca" pod "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" (UID: "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:53.533396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.533340 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-config" pod "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" (UID: "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:41:53.534777 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.534751 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-oauth-config" pod "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" (UID: "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:53.534895 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.534881 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-serving-cert" pod "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" (UID: "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:41:53.535007 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.534989 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-kube-api-access-b6rj8" pod "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" (UID: "8ecc4c47-82f4-40fb-8f8d-762d3af66a0d"). InnerVolumeSpecName "kube-api-access-b6rj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:41:53.633780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.633747 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6rj8\" (UniqueName: \"kubernetes.io/projected/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-kube-api-access-b6rj8\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:53.633780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.633776 4310 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-serving-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:53.633780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.633788 4310 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-config\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:53.634046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.633797 4310 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-service-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:53.634046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.633806 4310 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-console-oauth-config\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:53.634046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.633814 4310 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-trusted-ca-bundle\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:53.634046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:53.633823 4310 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d-oauth-serving-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:41:54.195568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:54.195536 4310 generic.go:356] "Generic (PLEG): container finished" podID="8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" containerID="a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f" exitCode=0 Nov 02 17:41:54.195568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:54.195572 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b76874c59-2mf6g" event={"ID":"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d","Type":"ContainerDied","Data":"a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f"} Nov 02 17:41:54.196021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:54.195587 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b76874c59-2mf6g" Nov 02 17:41:54.196021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:54.195605 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b76874c59-2mf6g" event={"ID":"8ecc4c47-82f4-40fb-8f8d-762d3af66a0d","Type":"ContainerDied","Data":"1a083402ff74c3e4a06ce8bdde932487e15f212456bbe5650e6237e062aa0597"} Nov 02 17:41:54.196021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:54.195624 4310 scope.go:122] "RemoveContainer" containerID="a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f" Nov 02 17:41:54.206542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:54.206515 4310 scope.go:122] "RemoveContainer" containerID="a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f" Nov 02 17:41:54.206870 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:41:54.206849 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f\": container with ID starting with a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f not found: ID does not exist" containerID="a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f" Nov 02 17:41:54.206948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:54.206876 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f"} err="failed to get container status \"a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f\": rpc error: code = NotFound desc = could not find container \"a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f\": container with ID starting with a7ef4187c2a3504acab37da3eef03a4159b2baa43ff77b43b231ac5eef6ea59f not found: ID does not exist" Nov 02 17:41:54.227465 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:54.227430 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b76874c59-2mf6g"] Nov 02 17:41:54.230154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:54.230128 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b76874c59-2mf6g"] Nov 02 17:41:54.671139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:41:54.671107 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" path="/var/lib/kubelet/pods/8ecc4c47-82f4-40fb-8f8d-762d3af66a0d/volumes" Nov 02 17:42:01.213648 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:42:01.213616 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:01.316253 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:42:01.316178 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:01.648890 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:42:01.648810 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:42:03.521505 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:42:03.521467 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:42:33.522244 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:42:33.522155 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:42:45.386624 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:42:45.386594 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:03.522378 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:43:03.522340 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:43:20.386173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:20.386138 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:22.165142 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:22.165101 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:22.812862 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:22.812828 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:23.054305 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:23.054271 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:24.808543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.808510 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5"] Nov 02 17:43:24.808920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.808758 4310 cpu_manager.go:414] "containerMap: removing container" podUID="8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" containerName="console" Nov 02 17:43:24.808920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.808768 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" containerName="console" Nov 02 17:43:24.808920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.808829 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ecc4c47-82f4-40fb-8f8d-762d3af66a0d" containerName="console" Nov 02 17:43:24.810585 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.810569 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:24.815171 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.815152 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-vww9b\"" Nov 02 17:43:24.817918 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.817899 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 17:43:24.817981 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.817945 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 17:43:24.824621 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.824601 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5"] Nov 02 17:43:24.903718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.903689 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w"] Nov 02 17:43:24.905714 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.905698 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:24.910590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.910569 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7jr\" (UniqueName: \"kubernetes.io/projected/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-kube-api-access-gs7jr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:24.910697 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.910605 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:24.910748 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.910704 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:24.913593 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.913570 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2"] Nov 02 17:43:24.915729 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.915714 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:24.918919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.918899 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-54bzl\"" Nov 02 17:43:24.923507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.923488 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w"] Nov 02 17:43:24.926391 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.926367 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 17:43:24.926491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.926377 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 17:43:24.926491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.926417 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 17:43:24.936639 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.936616 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:24.970450 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.970423 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2"] Nov 02 17:43:24.978133 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.978109 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 17:43:24.978464 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.978447 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d64z4\"" Nov 02 17:43:24.978608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.978593 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 17:43:24.978650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:24.978640 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 17:43:25.011139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011107 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:25.011317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011149 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:25.011317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011184 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.011317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011213 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.011317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011240 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:25.011506 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011379 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnc7x\" (UniqueName: \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-kube-api-access-gnc7x\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.011506 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011426 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7jr\" (UniqueName: \"kubernetes.io/projected/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-kube-api-access-gs7jr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:25.011506 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011445 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57vr9\" (UniqueName: \"kubernetes.io/projected/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-kube-api-access-57vr9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:25.011613 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011506 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:25.011613 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011599 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.011685 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011647 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:25.011735 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011686 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.011735 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011712 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.011825 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011777 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b795ce-68f6-4937-b1b7-c56b4c5919fe-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.011995 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.011975 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:25.022536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.022509 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7jr\" (UniqueName: \"kubernetes.io/projected/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-kube-api-access-gs7jr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:25.112248 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112156 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b795ce-68f6-4937-b1b7-c56b4c5919fe-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.112248 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112199 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:25.112248 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112222 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.112248 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112240 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.112248 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112256 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:25.112630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112381 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnc7x\" (UniqueName: \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-kube-api-access-gnc7x\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.112630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112480 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57vr9\" (UniqueName: \"kubernetes.io/projected/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-kube-api-access-57vr9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:25.112743 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112632 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:25.112743 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112655 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.112743 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112680 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:25.112743 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112731 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.112899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.112762 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.113172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.113149 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.114608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.114585 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b795ce-68f6-4937-b1b7-c56b4c5919fe-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.114755 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.114734 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.114873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.114857 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.114960 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.114946 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.122365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.122338 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:25.122849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.122828 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnc7x\" (UniqueName: \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-kube-api-access-gnc7x\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.122927 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.122853 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gwgc2\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.122974 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.122954 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57vr9\" (UniqueName: \"kubernetes.io/projected/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-kube-api-access-57vr9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:25.129647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.129621 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:25.218922 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.218881 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:25.225939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.225918 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:25.391241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.391163 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5"] Nov 02 17:43:25.393907 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:43:25.393884 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ad7d48_bb6c_4171_9e06_bcf6adb5d172.slice/crio-8e7bb43c1751169dfa065f4161d32126ee86060d327255a70d7abcc50da00190 WatchSource:0}: Error finding container 8e7bb43c1751169dfa065f4161d32126ee86060d327255a70d7abcc50da00190: Status 404 returned error can't find the container with id 8e7bb43c1751169dfa065f4161d32126ee86060d327255a70d7abcc50da00190 Nov 02 17:43:25.491069 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.491046 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w"] Nov 02 17:43:25.493237 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:43:25.493205 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c5fb4d_dd1c_4cc3_98f6_5a0d06315695.slice/crio-12e819808aae37ac2ead6e05ea4a4e98fc0e8c7f597b5eae5c955eb401131e6c WatchSource:0}: Error finding container 12e819808aae37ac2ead6e05ea4a4e98fc0e8c7f597b5eae5c955eb401131e6c: Status 404 returned error can't find the container with id 12e819808aae37ac2ead6e05ea4a4e98fc0e8c7f597b5eae5c955eb401131e6c Nov 02 17:43:25.495979 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.495954 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2"] Nov 02 17:43:25.499763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.499734 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" event={"ID":"60ad7d48-bb6c-4171-9e06-bcf6adb5d172","Type":"ContainerStarted","Data":"8e7bb43c1751169dfa065f4161d32126ee86060d327255a70d7abcc50da00190"} Nov 02 17:43:25.499883 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:43:25.499864 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b795ce_68f6_4937_b1b7_c56b4c5919fe.slice/crio-28ee84e1321b1516a729587682de29cc818bceb5c3a55af96c318c109cd03adb WatchSource:0}: Error finding container 28ee84e1321b1516a729587682de29cc818bceb5c3a55af96c318c109cd03adb: Status 404 returned error can't find the container with id 28ee84e1321b1516a729587682de29cc818bceb5c3a55af96c318c109cd03adb Nov 02 17:43:25.500854 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:25.500832 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" event={"ID":"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695","Type":"ContainerStarted","Data":"12e819808aae37ac2ead6e05ea4a4e98fc0e8c7f597b5eae5c955eb401131e6c"} Nov 02 17:43:26.180199 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:26.180114 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:26.506736 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:26.506659 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" event={"ID":"40b795ce-68f6-4937-b1b7-c56b4c5919fe","Type":"ContainerStarted","Data":"28ee84e1321b1516a729587682de29cc818bceb5c3a55af96c318c109cd03adb"} Nov 02 17:43:29.427345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:29.427309 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:43:29.427865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:29.427442 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:43:29.521292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:29.521256 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" event={"ID":"40b795ce-68f6-4937-b1b7-c56b4c5919fe","Type":"ContainerStarted","Data":"97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e"} Nov 02 17:43:29.521551 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:29.521532 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:29.546217 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:29.546161 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" podStartSLOduration=1.621447517 podStartE2EDuration="5.54614583s" totalImagesPullingTime="3.924698313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:29.544423823 +0000 UTC m=+206.987506495" watchObservedRunningTime="2026-11-02 17:43:29.54614583 +0000 UTC m=+206.989228466" Nov 02 17:43:29.951774 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:29.951735 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:30.058625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:30.058591 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:30.528659 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:30.528623 4310 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-gwgc2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:43:30.529142 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:30.528681 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" podUID="40b795ce-68f6-4937-b1b7-c56b4c5919fe" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:43:31.005118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:31.005057 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:31.116821 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:31.116740 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:32.168274 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:32.168210 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:32.535791 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:32.535696 4310 generic.go:356] "Generic (PLEG): container finished" podID="60ad7d48-bb6c-4171-9e06-bcf6adb5d172" containerID="f20be1a477bc16b9d457f3b1d5682c02fb80ce6476ac581ad1b2579b6ef308e5" exitCode=0 Nov 02 17:43:32.535938 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:32.535780 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" event={"ID":"60ad7d48-bb6c-4171-9e06-bcf6adb5d172","Type":"ContainerDied","Data":"f20be1a477bc16b9d457f3b1d5682c02fb80ce6476ac581ad1b2579b6ef308e5"} Nov 02 17:43:32.537355 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:32.537335 4310 generic.go:356] "Generic (PLEG): container finished" podID="39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" containerID="e8d69a4ff9a5a35518bb5658e60d05c42cc3b4e2abb3272944b54b5e8eee81cc" exitCode=0 Nov 02 17:43:32.537465 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:32.537377 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" event={"ID":"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695","Type":"ContainerDied","Data":"e8d69a4ff9a5a35518bb5658e60d05c42cc3b4e2abb3272944b54b5e8eee81cc"} Nov 02 17:43:33.527248 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:33.527223 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:43:33.529098 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:43:33.529055 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:43:33.533691 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:33.533669 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:35.550381 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:35.550348 4310 generic.go:356] "Generic (PLEG): container finished" podID="60ad7d48-bb6c-4171-9e06-bcf6adb5d172" containerID="8a1f0051fad05ea347e51b4059bd6495edc2f35d7bd27296884833077a76c179" exitCode=0 Nov 02 17:43:35.550846 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:35.550451 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" event={"ID":"60ad7d48-bb6c-4171-9e06-bcf6adb5d172","Type":"ContainerDied","Data":"8a1f0051fad05ea347e51b4059bd6495edc2f35d7bd27296884833077a76c179"} Nov 02 17:43:35.552237 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:35.552213 4310 generic.go:356] "Generic (PLEG): container finished" podID="39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" containerID="35ecdf7e7bf0dd6f4f4d126049c4c5ff679d8d7fab6c83b357f464296058c26e" exitCode=0 Nov 02 17:43:35.552338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:35.552265 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" event={"ID":"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695","Type":"ContainerDied","Data":"35ecdf7e7bf0dd6f4f4d126049c4c5ff679d8d7fab6c83b357f464296058c26e"} Nov 02 17:43:42.583620 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:42.583527 4310 generic.go:356] "Generic (PLEG): container finished" podID="60ad7d48-bb6c-4171-9e06-bcf6adb5d172" containerID="abc621ce282f70c9267be01569963d679aa5e6fbd552c832b9df7595efe89fff" exitCode=0 Nov 02 17:43:42.583996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:42.583606 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" event={"ID":"60ad7d48-bb6c-4171-9e06-bcf6adb5d172","Type":"ContainerDied","Data":"abc621ce282f70c9267be01569963d679aa5e6fbd552c832b9df7595efe89fff"} Nov 02 17:43:42.585728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:42.585697 4310 generic.go:356] "Generic (PLEG): container finished" podID="39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" containerID="755fde34438f01bba68cdc4720737a8453f3fa942d10e40387d0a628305b0b11" exitCode=0 Nov 02 17:43:42.585847 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:42.585778 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" event={"ID":"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695","Type":"ContainerDied","Data":"755fde34438f01bba68cdc4720737a8453f3fa942d10e40387d0a628305b0b11"} Nov 02 17:43:43.722308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.722284 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:43.742137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.742109 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:43.780048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.780017 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-util\" (UniqueName: \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-util\") pod \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " Nov 02 17:43:43.780246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.780064 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-kube-api-access-gs7jr\" (UniqueName: \"kubernetes.io/projected/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-kube-api-access-gs7jr\") pod \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " Nov 02 17:43:43.780246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.780102 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-bundle\" (UniqueName: \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-bundle\") pod \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\" (UID: \"60ad7d48-bb6c-4171-9e06-bcf6adb5d172\") " Nov 02 17:43:43.780571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.780542 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-bundle" pod "60ad7d48-bb6c-4171-9e06-bcf6adb5d172" (UID: "60ad7d48-bb6c-4171-9e06-bcf6adb5d172"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:43:43.782391 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.782366 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-kube-api-access-gs7jr" pod "60ad7d48-bb6c-4171-9e06-bcf6adb5d172" (UID: "60ad7d48-bb6c-4171-9e06-bcf6adb5d172"). InnerVolumeSpecName "kube-api-access-gs7jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:43.785674 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.785648 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-util" pod "60ad7d48-bb6c-4171-9e06-bcf6adb5d172" (UID: "60ad7d48-bb6c-4171-9e06-bcf6adb5d172"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:43:43.880671 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.880581 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-util\" (UniqueName: \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-util\") pod \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " Nov 02 17:43:43.880671 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.880660 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-bundle\" (UniqueName: \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-bundle\") pod \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " Nov 02 17:43:43.880888 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.880707 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-kube-api-access-57vr9\" (UniqueName: \"kubernetes.io/projected/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-kube-api-access-57vr9\") pod \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\" (UID: \"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695\") " Nov 02 17:43:43.880954 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.880923 4310 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-util\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:43.880954 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.880943 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs7jr\" (UniqueName: \"kubernetes.io/projected/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-kube-api-access-gs7jr\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:43.881035 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.880957 4310 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60ad7d48-bb6c-4171-9e06-bcf6adb5d172-bundle\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:43.881301 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.881268 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-bundle" pod "39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" (UID: "39c5fb4d-dd1c-4cc3-98f6-5a0d06315695"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:43:43.882752 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.882719 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-kube-api-access-57vr9" pod "39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" (UID: "39c5fb4d-dd1c-4cc3-98f6-5a0d06315695"). InnerVolumeSpecName "kube-api-access-57vr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:43.885253 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.885228 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-util" pod "39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" (UID: "39c5fb4d-dd1c-4cc3-98f6-5a0d06315695"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:43:43.982047 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.981991 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57vr9\" (UniqueName: \"kubernetes.io/projected/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-kube-api-access-57vr9\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:43.982047 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.982038 4310 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-util\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:43.982047 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:43.982052 4310 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39c5fb4d-dd1c-4cc3-98f6-5a0d06315695-bundle\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:43:44.595026 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:44.594992 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" Nov 02 17:43:44.595246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:44.594990 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644glqc5" event={"ID":"60ad7d48-bb6c-4171-9e06-bcf6adb5d172","Type":"ContainerDied","Data":"8e7bb43c1751169dfa065f4161d32126ee86060d327255a70d7abcc50da00190"} Nov 02 17:43:44.595246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:44.595169 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e7bb43c1751169dfa065f4161d32126ee86060d327255a70d7abcc50da00190" Nov 02 17:43:44.596909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:44.596888 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" event={"ID":"39c5fb4d-dd1c-4cc3-98f6-5a0d06315695","Type":"ContainerDied","Data":"12e819808aae37ac2ead6e05ea4a4e98fc0e8c7f597b5eae5c955eb401131e6c"} Nov 02 17:43:44.597013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:44.596911 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxtp2w" Nov 02 17:43:44.597013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:44.596922 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12e819808aae37ac2ead6e05ea4a4e98fc0e8c7f597b5eae5c955eb401131e6c" Nov 02 17:43:45.458006 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:45.457928 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:52.382311 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:52.382280 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:52.924885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:52.924855 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:56.973679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:56.973640 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:57.123442 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:57.123405 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:59.500651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:59.500610 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:43:59.894693 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:43:59.894614 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:00.300882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:00.300850 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:01.001943 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.001911 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:01.325128 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325042 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw"] Nov 02 17:44:01.325325 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325313 4310 cpu_manager.go:414] "containerMap: removing container" podUID="60ad7d48-bb6c-4171-9e06-bcf6adb5d172" containerName="pull" Nov 02 17:44:01.325364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325327 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ad7d48-bb6c-4171-9e06-bcf6adb5d172" containerName="pull" Nov 02 17:44:01.325364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325335 4310 cpu_manager.go:414] "containerMap: removing container" podUID="60ad7d48-bb6c-4171-9e06-bcf6adb5d172" containerName="extract" Nov 02 17:44:01.325364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325340 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ad7d48-bb6c-4171-9e06-bcf6adb5d172" containerName="extract" Nov 02 17:44:01.325364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325349 4310 cpu_manager.go:414] "containerMap: removing container" podUID="39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" containerName="util" Nov 02 17:44:01.325364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325356 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" containerName="util" Nov 02 17:44:01.325511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325368 4310 cpu_manager.go:414] "containerMap: removing container" podUID="39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" containerName="extract" Nov 02 17:44:01.325511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325373 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" containerName="extract" Nov 02 17:44:01.325511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325385 4310 cpu_manager.go:414] "containerMap: removing container" podUID="39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" containerName="pull" Nov 02 17:44:01.325511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325390 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" containerName="pull" Nov 02 17:44:01.325511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325400 4310 cpu_manager.go:414] "containerMap: removing container" podUID="60ad7d48-bb6c-4171-9e06-bcf6adb5d172" containerName="util" Nov 02 17:44:01.325511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325405 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ad7d48-bb6c-4171-9e06-bcf6adb5d172" containerName="util" Nov 02 17:44:01.325511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325451 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ad7d48-bb6c-4171-9e06-bcf6adb5d172" containerName="extract" Nov 02 17:44:01.325511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.325461 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c5fb4d-dd1c-4cc3-98f6-5a0d06315695" containerName="extract" Nov 02 17:44:01.331447 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.331422 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.380772 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.380735 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 17:44:01.380989 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.380969 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 17:44:01.381327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.381311 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nkllc\"" Nov 02 17:44:01.381398 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.381384 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 17:44:01.382603 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.382592 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 17:44:01.382675 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.382663 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 17:44:01.415944 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.415912 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw"] Nov 02 17:44:01.422560 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.422534 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8329f1cc-0f29-4f30-9597-7a50d819d14a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.422686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.422580 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8329f1cc-0f29-4f30-9597-7a50d819d14a-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.422686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.422608 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8329f1cc-0f29-4f30-9597-7a50d819d14a-manifests\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.422686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.422671 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8329f1cc-0f29-4f30-9597-7a50d819d14a-tmp\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.422814 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.422799 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8329f1cc-0f29-4f30-9597-7a50d819d14a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.422853 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.422833 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc8wx\" (UniqueName: \"kubernetes.io/projected/8329f1cc-0f29-4f30-9597-7a50d819d14a-kube-api-access-lc8wx\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.523736 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.523696 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8329f1cc-0f29-4f30-9597-7a50d819d14a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.523933 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.523754 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8329f1cc-0f29-4f30-9597-7a50d819d14a-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.523933 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.523786 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8329f1cc-0f29-4f30-9597-7a50d819d14a-manifests\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.523933 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.523803 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8329f1cc-0f29-4f30-9597-7a50d819d14a-tmp\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.523933 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.523824 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8329f1cc-0f29-4f30-9597-7a50d819d14a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.523933 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.523863 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc8wx\" (UniqueName: \"kubernetes.io/projected/8329f1cc-0f29-4f30-9597-7a50d819d14a-kube-api-access-lc8wx\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.524433 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.524399 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8329f1cc-0f29-4f30-9597-7a50d819d14a-manifests\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.526102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.526066 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8329f1cc-0f29-4f30-9597-7a50d819d14a-tmp\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.526413 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.526390 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8329f1cc-0f29-4f30-9597-7a50d819d14a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.526667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.526647 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8329f1cc-0f29-4f30-9597-7a50d819d14a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.526667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.526660 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8329f1cc-0f29-4f30-9597-7a50d819d14a-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.543745 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.543718 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc8wx\" (UniqueName: \"kubernetes.io/projected/8329f1cc-0f29-4f30-9597-7a50d819d14a-kube-api-access-lc8wx\") pod \"opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw\" (UID: \"8329f1cc-0f29-4f30-9597-7a50d819d14a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.602871 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.602774 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:01.643868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.643832 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:01.946580 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:01.946489 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:02.063016 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:44:02.062988 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8329f1cc_0f29_4f30_9597_7a50d819d14a.slice/crio-7b476a35783e7064e994388118db5371c48220148d4e3fac8ab5d76a1b0b9880 WatchSource:0}: Error finding container 7b476a35783e7064e994388118db5371c48220148d4e3fac8ab5d76a1b0b9880: Status 404 returned error can't find the container with id 7b476a35783e7064e994388118db5371c48220148d4e3fac8ab5d76a1b0b9880 Nov 02 17:44:02.075413 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:02.075382 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw"] Nov 02 17:44:02.227597 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:02.227557 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:02.567627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:02.567553 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:02.664574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:02.664536 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" event={"ID":"8329f1cc-0f29-4f30-9597-7a50d819d14a","Type":"ContainerStarted","Data":"7b476a35783e7064e994388118db5371c48220148d4e3fac8ab5d76a1b0b9880"} Nov 02 17:44:02.814332 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:02.814297 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:03.060241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:03.060193 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:03.282175 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:03.282121 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:03.530399 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:44:03.530359 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:44:03.803252 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:03.803163 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:05.677916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:05.677835 4310 generic.go:356] "Generic (PLEG): container finished" podID="8329f1cc-0f29-4f30-9597-7a50d819d14a" containerID="71e59a2d58943882919adb5b304f223787ffd718794053a6f7189a27ab9217fc" exitCode=0 Nov 02 17:44:05.677916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:05.677878 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" event={"ID":"8329f1cc-0f29-4f30-9597-7a50d819d14a","Type":"ContainerDied","Data":"71e59a2d58943882919adb5b304f223787ffd718794053a6f7189a27ab9217fc"} Nov 02 17:44:06.684582 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:06.684543 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" event={"ID":"8329f1cc-0f29-4f30-9597-7a50d819d14a","Type":"ContainerStarted","Data":"dcdb44625a8ddae2caac7d24ec66db18a0a4788d9a6f7ddd155bac5cdc1bc09f"} Nov 02 17:44:06.685059 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:06.684634 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:06.783529 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:06.783475 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" podStartSLOduration=2.096726533 podStartE2EDuration="5.78345993s" totalImagesPullingTime="3.230080639s" totalInitContainerRuntime="456.652758ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:06.775902799 +0000 UTC m=+244.218985436" watchObservedRunningTime="2026-11-02 17:44:06.78345993 +0000 UTC m=+244.226542567" Nov 02 17:44:07.486466 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:07.486420 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:08.013647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:08.013613 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:08.864294 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:08.864262 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:09.195324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:09.195219 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:10.095412 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:10.095375 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:10.838785 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:10.838742 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:14.820262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:14.820229 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:14.996609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:14.996575 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:16.549841 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:16.549766 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:16.741505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:16.741473 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:17.690918 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:17.690887 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw" Nov 02 17:44:18.806280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:18.806247 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:18.960125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:18.960089 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:19.444843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:19.444807 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:19.713215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:19.713181 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:19.991203 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:19.991108 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:20.320202 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:20.320117 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:21.700459 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:21.700425 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:24.144866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:24.144835 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:24.479483 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:24.479438 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:29.796861 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:29.796811 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:29.911824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:29.911767 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:30.192582 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:30.192467 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:30.337697 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:30.337664 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:30.558622 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:30.558589 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:30.675409 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:30.675372 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:33.522713 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:44:33.522680 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:44:35.173634 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.173574 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:35.423069 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.423032 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:35.537583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.537544 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x"] Nov 02 17:44:35.541013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.540989 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" Nov 02 17:44:35.550432 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.550409 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 17:44:35.551904 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.551880 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 17:44:35.552041 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.551973 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c45v5\"" Nov 02 17:44:35.565764 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.565734 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x"] Nov 02 17:44:35.586527 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.586493 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvdjx\" (UniqueName: \"kubernetes.io/projected/4de65c0e-62d2-4e29-b13f-309a0ce43deb-kube-api-access-fvdjx\") pod \"kube-auth-proxy-555d4cc554-ssg7x\" (UID: \"4de65c0e-62d2-4e29-b13f-309a0ce43deb\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" Nov 02 17:44:35.586678 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.586644 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de65c0e-62d2-4e29-b13f-309a0ce43deb-tmp\") pod \"kube-auth-proxy-555d4cc554-ssg7x\" (UID: \"4de65c0e-62d2-4e29-b13f-309a0ce43deb\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" Nov 02 17:44:35.586725 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.586685 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de65c0e-62d2-4e29-b13f-309a0ce43deb-tls-certs\") pod \"kube-auth-proxy-555d4cc554-ssg7x\" (UID: \"4de65c0e-62d2-4e29-b13f-309a0ce43deb\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" Nov 02 17:44:35.661158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.661130 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:35.687290 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.687248 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de65c0e-62d2-4e29-b13f-309a0ce43deb-tmp\") pod \"kube-auth-proxy-555d4cc554-ssg7x\" (UID: \"4de65c0e-62d2-4e29-b13f-309a0ce43deb\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" Nov 02 17:44:35.687290 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.687293 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de65c0e-62d2-4e29-b13f-309a0ce43deb-tls-certs\") pod \"kube-auth-proxy-555d4cc554-ssg7x\" (UID: \"4de65c0e-62d2-4e29-b13f-309a0ce43deb\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" Nov 02 17:44:35.687555 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.687365 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvdjx\" (UniqueName: \"kubernetes.io/projected/4de65c0e-62d2-4e29-b13f-309a0ce43deb-kube-api-access-fvdjx\") pod \"kube-auth-proxy-555d4cc554-ssg7x\" (UID: \"4de65c0e-62d2-4e29-b13f-309a0ce43deb\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" Nov 02 17:44:35.689566 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.689539 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de65c0e-62d2-4e29-b13f-309a0ce43deb-tmp\") pod \"kube-auth-proxy-555d4cc554-ssg7x\" (UID: \"4de65c0e-62d2-4e29-b13f-309a0ce43deb\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" Nov 02 17:44:35.689898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.689879 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de65c0e-62d2-4e29-b13f-309a0ce43deb-tls-certs\") pod \"kube-auth-proxy-555d4cc554-ssg7x\" (UID: \"4de65c0e-62d2-4e29-b13f-309a0ce43deb\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" Nov 02 17:44:35.708362 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.708337 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvdjx\" (UniqueName: \"kubernetes.io/projected/4de65c0e-62d2-4e29-b13f-309a0ce43deb-kube-api-access-fvdjx\") pod \"kube-auth-proxy-555d4cc554-ssg7x\" (UID: \"4de65c0e-62d2-4e29-b13f-309a0ce43deb\") " pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" Nov 02 17:44:35.855587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.855505 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" Nov 02 17:44:35.955883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:35.955844 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:36.150055 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:36.150025 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x"] Nov 02 17:44:36.152802 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:44:36.152770 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de65c0e_62d2_4e29_b13f_309a0ce43deb.slice/crio-64f3dfd2ee5134c1fbf172a87b502243d3b27bcfbe0808556f4177910b1ecc96 WatchSource:0}: Error finding container 64f3dfd2ee5134c1fbf172a87b502243d3b27bcfbe0808556f4177910b1ecc96: Status 404 returned error can't find the container with id 64f3dfd2ee5134c1fbf172a87b502243d3b27bcfbe0808556f4177910b1ecc96 Nov 02 17:44:36.213914 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:36.213876 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:36.411883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:36.411801 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:36.681402 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:36.681316 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:36.802876 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:36.802842 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" event={"ID":"4de65c0e-62d2-4e29-b13f-309a0ce43deb","Type":"ContainerStarted","Data":"64f3dfd2ee5134c1fbf172a87b502243d3b27bcfbe0808556f4177910b1ecc96"} Nov 02 17:44:36.879419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:36.879386 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:37.038846 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:37.038780 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:37.278507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:37.278430 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:37.816022 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:37.815986 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:38.263930 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:38.263895 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:38.567437 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:38.567359 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:38.813406 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:38.813376 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" event={"ID":"4de65c0e-62d2-4e29-b13f-309a0ce43deb","Type":"ContainerStarted","Data":"f725e6b5e731432fdbc98d86d2bf14bc5a056f8427c7ee5314b34244675f2232"} Nov 02 17:44:38.835822 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:38.835734 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-555d4cc554-ssg7x" podStartSLOduration=1.311634352 podStartE2EDuration="3.835720718s" totalImagesPullingTime="2.524086366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:38.835566987 +0000 UTC m=+276.278649624" watchObservedRunningTime="2026-11-02 17:44:38.835720718 +0000 UTC m=+276.278803354" Nov 02 17:44:39.506969 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.506933 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx"] Nov 02 17:44:39.510285 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.510264 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.514006 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.513979 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 17:44:39.514162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.513996 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-d2zhc\"" Nov 02 17:44:39.514907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.514889 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 17:44:39.517848 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.517827 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 17:44:39.534205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.534181 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx"] Nov 02 17:44:39.628513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.628476 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af9c459a-3726-41aa-b94d-02bdcfd9d10e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.628930 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.628525 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af9c459a-3726-41aa-b94d-02bdcfd9d10e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.628930 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.628583 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af9c459a-3726-41aa-b94d-02bdcfd9d10e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.628930 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.628652 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jqwn\" (UniqueName: \"kubernetes.io/projected/af9c459a-3726-41aa-b94d-02bdcfd9d10e-kube-api-access-7jqwn\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.729323 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.729286 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jqwn\" (UniqueName: \"kubernetes.io/projected/af9c459a-3726-41aa-b94d-02bdcfd9d10e-kube-api-access-7jqwn\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.729524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.729333 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af9c459a-3726-41aa-b94d-02bdcfd9d10e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.729524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.729360 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af9c459a-3726-41aa-b94d-02bdcfd9d10e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.729524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.729379 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af9c459a-3726-41aa-b94d-02bdcfd9d10e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.729726 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.729705 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af9c459a-3726-41aa-b94d-02bdcfd9d10e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.729775 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.729764 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af9c459a-3726-41aa-b94d-02bdcfd9d10e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.730189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.730171 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af9c459a-3726-41aa-b94d-02bdcfd9d10e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.738574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.738540 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jqwn\" (UniqueName: \"kubernetes.io/projected/af9c459a-3726-41aa-b94d-02bdcfd9d10e-kube-api-access-7jqwn\") pod \"kserve-module-controller-manager-7566cf5fc8-rxjvx\" (UID: \"af9c459a-3726-41aa-b94d-02bdcfd9d10e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:39.822761 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:39.822688 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:40.118919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:40.118839 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx"] Nov 02 17:44:40.121762 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:44:40.121735 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9c459a_3726_41aa_b94d_02bdcfd9d10e.slice/crio-dd64cb42fa7b48d2208732dded98c84da808fa8361f2fb8d72dedb55c2be57e3 WatchSource:0}: Error finding container dd64cb42fa7b48d2208732dded98c84da808fa8361f2fb8d72dedb55c2be57e3: Status 404 returned error can't find the container with id dd64cb42fa7b48d2208732dded98c84da808fa8361f2fb8d72dedb55c2be57e3 Nov 02 17:44:40.824597 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:40.824551 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" event={"ID":"af9c459a-3726-41aa-b94d-02bdcfd9d10e","Type":"ContainerStarted","Data":"dd64cb42fa7b48d2208732dded98c84da808fa8361f2fb8d72dedb55c2be57e3"} Nov 02 17:44:41.830571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:41.830529 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" event={"ID":"af9c459a-3726-41aa-b94d-02bdcfd9d10e","Type":"ContainerStarted","Data":"6a8af60b0f3e5f3da5134838604f5f87f1c38f1792074d79b4b7b2ec40da7d1b"} Nov 02 17:44:41.830943 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:41.830655 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:44:41.853131 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:41.853059 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" podStartSLOduration=1.298556082 podStartE2EDuration="2.853046098s" totalImagesPullingTime="1.554490016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:41.851248319 +0000 UTC m=+279.294330955" watchObservedRunningTime="2026-11-02 17:44:41.853046098 +0000 UTC m=+279.296128739" Nov 02 17:44:42.431112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:42.431044 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:42.531173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:42.531140 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:45.435065 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:45.435028 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:45.559908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:45.559871 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:50.654691 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:50.654657 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:50.771206 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:50.771172 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:51.057025 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:51.056989 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:51.158904 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:51.158873 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:51.279911 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:51.279875 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:51.397784 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:51.397690 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:58.236938 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:58.236901 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:58.334227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:58.334200 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:58.629924 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:58.629847 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:44:58.731306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:44:58.731270 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:02.836219 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:02.836189 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rxjvx" Nov 02 17:45:03.522172 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:45:03.522139 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:45:04.049698 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:04.049637 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:04.149224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:04.149192 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:07.054234 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.054201 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:07.168493 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.168459 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:07.448805 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.448733 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z2pw2"] Nov 02 17:45:07.452198 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.452182 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:07.460264 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.460244 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 17:45:07.460348 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.460315 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 17:45:07.461228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.461207 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 17:45:07.463415 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.463395 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z2pw2"] Nov 02 17:45:07.470653 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.470631 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:07.523280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.523250 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j8t8g\"" Nov 02 17:45:07.551203 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.551173 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z2pw2"] Nov 02 17:45:07.572555 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.572526 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bgxh\" (UniqueName: \"kubernetes.io/projected/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-kube-api-access-7bgxh\") pod \"limitador-limitador-74dfcfb894-z2pw2\" (UID: \"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:07.572649 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.572609 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-config-file\") pod \"limitador-limitador-74dfcfb894-z2pw2\" (UID: \"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:07.673952 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.673917 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-config-file\") pod \"limitador-limitador-74dfcfb894-z2pw2\" (UID: \"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:07.674151 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.673982 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bgxh\" (UniqueName: \"kubernetes.io/projected/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-kube-api-access-7bgxh\") pod \"limitador-limitador-74dfcfb894-z2pw2\" (UID: \"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:07.674611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.674589 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-config-file\") pod \"limitador-limitador-74dfcfb894-z2pw2\" (UID: \"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:07.686404 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.686368 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bgxh\" (UniqueName: \"kubernetes.io/projected/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-kube-api-access-7bgxh\") pod \"limitador-limitador-74dfcfb894-z2pw2\" (UID: \"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:07.765127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:07.765095 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:08.036790 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:08.036721 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z2pw2"] Nov 02 17:45:08.040182 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:45:08.040155 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb4334e_c9bd_4c5f_b193_2ef122ba7c63.slice/crio-3b7e1a3b4bfdcd26f4bac017ff69975dba10a3cb0282eba6d5aa66149a1490cd WatchSource:0}: Error finding container 3b7e1a3b4bfdcd26f4bac017ff69975dba10a3cb0282eba6d5aa66149a1490cd: Status 404 returned error can't find the container with id 3b7e1a3b4bfdcd26f4bac017ff69975dba10a3cb0282eba6d5aa66149a1490cd Nov 02 17:45:08.042453 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:08.042438 4310 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:45:08.934594 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:08.934561 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" event={"ID":"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63","Type":"ContainerStarted","Data":"3b7e1a3b4bfdcd26f4bac017ff69975dba10a3cb0282eba6d5aa66149a1490cd"} Nov 02 17:45:11.949658 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:11.949617 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" event={"ID":"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63","Type":"ContainerStarted","Data":"6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd"} Nov 02 17:45:11.950139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:11.949718 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:11.967673 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:11.967614 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" podStartSLOduration=1.22300174 podStartE2EDuration="4.967599358s" totalImagesPullingTime="3.744597618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:11.966602651 +0000 UTC m=+309.409685290" watchObservedRunningTime="2026-11-02 17:45:11.967599358 +0000 UTC m=+309.410681994" Nov 02 17:45:18.715620 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:18.715496 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:21.348822 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.348781 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z2pw2"] Nov 02 17:45:21.349461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.348980 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" podUID="4eb4334e-c9bd-4c5f-b193-2ef122ba7c63" containerName="limitador" containerID="cri-o://6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd" gracePeriod=30 Nov 02 17:45:21.349852 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.349720 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:21.894367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.894340 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:21.991515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.991475 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-config-file\" (UniqueName: \"kubernetes.io/configmap/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-config-file\") pod \"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63\" (UID: \"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63\") " Nov 02 17:45:21.991716 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.991619 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-kube-api-access-7bgxh\" (UniqueName: \"kubernetes.io/projected/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-kube-api-access-7bgxh\") pod \"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63\" (UID: \"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63\") " Nov 02 17:45:21.991900 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.991878 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-config-file" pod "4eb4334e-c9bd-4c5f-b193-2ef122ba7c63" (UID: "4eb4334e-c9bd-4c5f-b193-2ef122ba7c63"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:21.992746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.992718 4310 generic.go:356] "Generic (PLEG): container finished" podID="4eb4334e-c9bd-4c5f-b193-2ef122ba7c63" containerID="6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd" exitCode=0 Nov 02 17:45:21.992835 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.992769 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" Nov 02 17:45:21.992835 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.992806 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" event={"ID":"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63","Type":"ContainerDied","Data":"6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd"} Nov 02 17:45:21.992905 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.992867 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z2pw2" event={"ID":"4eb4334e-c9bd-4c5f-b193-2ef122ba7c63","Type":"ContainerDied","Data":"3b7e1a3b4bfdcd26f4bac017ff69975dba10a3cb0282eba6d5aa66149a1490cd"} Nov 02 17:45:21.992935 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.992902 4310 scope.go:122] "RemoveContainer" containerID="6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd" Nov 02 17:45:21.994028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:21.994009 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-kube-api-access-7bgxh" pod "4eb4334e-c9bd-4c5f-b193-2ef122ba7c63" (UID: "4eb4334e-c9bd-4c5f-b193-2ef122ba7c63"). InnerVolumeSpecName "kube-api-access-7bgxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:22.002810 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:22.002783 4310 scope.go:122] "RemoveContainer" containerID="6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd" Nov 02 17:45:22.003108 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:45:22.003087 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd\": container with ID starting with 6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd not found: ID does not exist" containerID="6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd" Nov 02 17:45:22.003170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:22.003119 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd"} err="failed to get container status \"6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd\": rpc error: code = NotFound desc = could not find container \"6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd\": container with ID starting with 6a9a198e37856737650d37475c5a127d68747eb78fbc7790ecf452e7c7916bcd not found: ID does not exist" Nov 02 17:45:22.092825 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:22.092793 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bgxh\" (UniqueName: \"kubernetes.io/projected/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-kube-api-access-7bgxh\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:22.092825 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:22.092822 4310 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63-config-file\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:22.314494 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:22.314463 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z2pw2"] Nov 02 17:45:22.318863 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:22.318837 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z2pw2"] Nov 02 17:45:22.671312 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:22.671233 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb4334e-c9bd-4c5f-b193-2ef122ba7c63" path="/var/lib/kubelet/pods/4eb4334e-c9bd-4c5f-b193-2ef122ba7c63/volumes" Nov 02 17:45:23.481817 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:23.481785 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:24.130681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.130643 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7"] Nov 02 17:45:24.131100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.130945 4310 cpu_manager.go:414] "containerMap: removing container" podUID="4eb4334e-c9bd-4c5f-b193-2ef122ba7c63" containerName="limitador" Nov 02 17:45:24.131100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.130955 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb4334e-c9bd-4c5f-b193-2ef122ba7c63" containerName="limitador" Nov 02 17:45:24.131100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.131013 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb4334e-c9bd-4c5f-b193-2ef122ba7c63" containerName="limitador" Nov 02 17:45:24.135184 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.135163 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" Nov 02 17:45:24.142204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.142181 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 17:45:24.142694 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.142449 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 17:45:24.143861 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.143839 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7"] Nov 02 17:45:24.193009 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.192983 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 17:45:24.193009 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.192999 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-699ps\"" Nov 02 17:45:24.210400 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.210364 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1956a285-a1d9-4dd6-aa86-fa73ec4272fc-data\") pod \"postgres-7cdbf6dc44-g4jn7\" (UID: \"1956a285-a1d9-4dd6-aa86-fa73ec4272fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" Nov 02 17:45:24.210531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.210407 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4ht\" (UniqueName: \"kubernetes.io/projected/1956a285-a1d9-4dd6-aa86-fa73ec4272fc-kube-api-access-gg4ht\") pod \"postgres-7cdbf6dc44-g4jn7\" (UID: \"1956a285-a1d9-4dd6-aa86-fa73ec4272fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" Nov 02 17:45:24.311449 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.311406 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1956a285-a1d9-4dd6-aa86-fa73ec4272fc-data\") pod \"postgres-7cdbf6dc44-g4jn7\" (UID: \"1956a285-a1d9-4dd6-aa86-fa73ec4272fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" Nov 02 17:45:24.311642 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.311454 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4ht\" (UniqueName: \"kubernetes.io/projected/1956a285-a1d9-4dd6-aa86-fa73ec4272fc-kube-api-access-gg4ht\") pod \"postgres-7cdbf6dc44-g4jn7\" (UID: \"1956a285-a1d9-4dd6-aa86-fa73ec4272fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" Nov 02 17:45:24.311806 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.311780 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1956a285-a1d9-4dd6-aa86-fa73ec4272fc-data\") pod \"postgres-7cdbf6dc44-g4jn7\" (UID: \"1956a285-a1d9-4dd6-aa86-fa73ec4272fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" Nov 02 17:45:24.320103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.320062 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4ht\" (UniqueName: \"kubernetes.io/projected/1956a285-a1d9-4dd6-aa86-fa73ec4272fc-kube-api-access-gg4ht\") pod \"postgres-7cdbf6dc44-g4jn7\" (UID: \"1956a285-a1d9-4dd6-aa86-fa73ec4272fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" Nov 02 17:45:24.448265 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.448178 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" Nov 02 17:45:24.740426 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:24.740391 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7"] Nov 02 17:45:24.744291 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:45:24.744265 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1956a285_a1d9_4dd6_aa86_fa73ec4272fc.slice/crio-b7ac44d91005cd25bbf9d12c31cceaf96696c63a407041336af60aad2ba247b9 WatchSource:0}: Error finding container b7ac44d91005cd25bbf9d12c31cceaf96696c63a407041336af60aad2ba247b9: Status 404 returned error can't find the container with id b7ac44d91005cd25bbf9d12c31cceaf96696c63a407041336af60aad2ba247b9 Nov 02 17:45:25.006778 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:25.006693 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" event={"ID":"1956a285-a1d9-4dd6-aa86-fa73ec4272fc","Type":"ContainerStarted","Data":"b7ac44d91005cd25bbf9d12c31cceaf96696c63a407041336af60aad2ba247b9"} Nov 02 17:45:28.630349 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:28.630309 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:28.740967 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:28.740929 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:31.033865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:31.033827 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" event={"ID":"1956a285-a1d9-4dd6-aa86-fa73ec4272fc","Type":"ContainerStarted","Data":"2ecd3db1b5bd2b3e9458cc59ce3657a89f863be9b77af2b40702da17db5e4b33"} Nov 02 17:45:31.034272 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:31.033895 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" Nov 02 17:45:31.057676 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:31.057603 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" podStartSLOduration=1.546747986 podStartE2EDuration="7.057581337s" totalImagesPullingTime="5.510833351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:31.05689343 +0000 UTC m=+328.499976067" watchObservedRunningTime="2026-11-02 17:45:31.057581337 +0000 UTC m=+328.500663973" Nov 02 17:45:32.297907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.297872 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:32.404148 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.404116 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4pwjv"] Nov 02 17:45:32.407212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.407190 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" Nov 02 17:45:32.413989 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.413804 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 17:45:32.413989 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.413890 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-96m7q\"" Nov 02 17:45:32.421689 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.421663 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4pwjv"] Nov 02 17:45:32.430409 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.430389 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:32.441836 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.441812 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5x726"] Nov 02 17:45:32.444858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.444839 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-5x726" Nov 02 17:45:32.456281 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.456258 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 17:45:32.456600 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.456346 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-bjjxs\"" Nov 02 17:45:32.458348 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.458326 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5x726"] Nov 02 17:45:32.481373 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.481348 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-5ncdw"] Nov 02 17:45:32.484625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.484605 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:32.488581 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.488562 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94716789-2a04-4aa1-a10e-b6e2fb1517e5-tls-certs\") pod \"model-serving-api-75bd89fdfc-5x726\" (UID: \"94716789-2a04-4aa1-a10e-b6e2fb1517e5\") " pod="opendatahub/model-serving-api-75bd89fdfc-5x726" Nov 02 17:45:32.488665 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.488593 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ct56\" (UniqueName: \"kubernetes.io/projected/94716789-2a04-4aa1-a10e-b6e2fb1517e5-kube-api-access-6ct56\") pod \"model-serving-api-75bd89fdfc-5x726\" (UID: \"94716789-2a04-4aa1-a10e-b6e2fb1517e5\") " pod="opendatahub/model-serving-api-75bd89fdfc-5x726" Nov 02 17:45:32.488665 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.488618 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05242e7d-02d8-4b62-a200-cdd9f108e64a-cert\") pod \"kserve-controller-manager-776cc66d47-4pwjv\" (UID: \"05242e7d-02d8-4b62-a200-cdd9f108e64a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" Nov 02 17:45:32.488775 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.488756 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppp9b\" (UniqueName: \"kubernetes.io/projected/05242e7d-02d8-4b62-a200-cdd9f108e64a-kube-api-access-ppp9b\") pod \"kserve-controller-manager-776cc66d47-4pwjv\" (UID: \"05242e7d-02d8-4b62-a200-cdd9f108e64a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" Nov 02 17:45:32.495244 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.495226 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5ncdw"] Nov 02 17:45:32.512688 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.512651 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 17:45:32.512896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.512878 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-q28zm\"" Nov 02 17:45:32.512965 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.512879 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 17:45:32.534424 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.534389 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:32.589507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.589428 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp9b\" (UniqueName: \"kubernetes.io/projected/05242e7d-02d8-4b62-a200-cdd9f108e64a-kube-api-access-ppp9b\") pod \"kserve-controller-manager-776cc66d47-4pwjv\" (UID: \"05242e7d-02d8-4b62-a200-cdd9f108e64a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" Nov 02 17:45:32.589507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.589500 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94716789-2a04-4aa1-a10e-b6e2fb1517e5-tls-certs\") pod \"model-serving-api-75bd89fdfc-5x726\" (UID: \"94716789-2a04-4aa1-a10e-b6e2fb1517e5\") " pod="opendatahub/model-serving-api-75bd89fdfc-5x726" Nov 02 17:45:32.589708 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.589538 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ct56\" (UniqueName: \"kubernetes.io/projected/94716789-2a04-4aa1-a10e-b6e2fb1517e5-kube-api-access-6ct56\") pod \"model-serving-api-75bd89fdfc-5x726\" (UID: \"94716789-2a04-4aa1-a10e-b6e2fb1517e5\") " pod="opendatahub/model-serving-api-75bd89fdfc-5x726" Nov 02 17:45:32.589708 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.589569 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05242e7d-02d8-4b62-a200-cdd9f108e64a-cert\") pod \"kserve-controller-manager-776cc66d47-4pwjv\" (UID: \"05242e7d-02d8-4b62-a200-cdd9f108e64a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" Nov 02 17:45:32.589708 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.589608 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abdb4611-dc86-47e3-997a-afb58e26fade-cert\") pod \"odh-model-controller-769875db56-5ncdw\" (UID: \"abdb4611-dc86-47e3-997a-afb58e26fade\") " pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:32.589708 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.589631 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz2gd\" (UniqueName: \"kubernetes.io/projected/abdb4611-dc86-47e3-997a-afb58e26fade-kube-api-access-jz2gd\") pod \"odh-model-controller-769875db56-5ncdw\" (UID: \"abdb4611-dc86-47e3-997a-afb58e26fade\") " pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:32.589708 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.589664 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdb4611-dc86-47e3-997a-afb58e26fade-metrics-certs\") pod \"odh-model-controller-769875db56-5ncdw\" (UID: \"abdb4611-dc86-47e3-997a-afb58e26fade\") " pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:32.592120 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.592070 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94716789-2a04-4aa1-a10e-b6e2fb1517e5-tls-certs\") pod \"model-serving-api-75bd89fdfc-5x726\" (UID: \"94716789-2a04-4aa1-a10e-b6e2fb1517e5\") " pod="opendatahub/model-serving-api-75bd89fdfc-5x726" Nov 02 17:45:32.592266 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.592205 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05242e7d-02d8-4b62-a200-cdd9f108e64a-cert\") pod \"kserve-controller-manager-776cc66d47-4pwjv\" (UID: \"05242e7d-02d8-4b62-a200-cdd9f108e64a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" Nov 02 17:45:32.599829 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.599798 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppp9b\" (UniqueName: \"kubernetes.io/projected/05242e7d-02d8-4b62-a200-cdd9f108e64a-kube-api-access-ppp9b\") pod \"kserve-controller-manager-776cc66d47-4pwjv\" (UID: \"05242e7d-02d8-4b62-a200-cdd9f108e64a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" Nov 02 17:45:32.600028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.600007 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ct56\" (UniqueName: \"kubernetes.io/projected/94716789-2a04-4aa1-a10e-b6e2fb1517e5-kube-api-access-6ct56\") pod \"model-serving-api-75bd89fdfc-5x726\" (UID: \"94716789-2a04-4aa1-a10e-b6e2fb1517e5\") " pod="opendatahub/model-serving-api-75bd89fdfc-5x726" Nov 02 17:45:32.638758 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.638717 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh"] Nov 02 17:45:32.642440 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.642417 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.662614 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.662579 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:32.662854 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.662831 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh"] Nov 02 17:45:32.690796 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.690766 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abdb4611-dc86-47e3-997a-afb58e26fade-cert\") pod \"odh-model-controller-769875db56-5ncdw\" (UID: \"abdb4611-dc86-47e3-997a-afb58e26fade\") " pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:32.690980 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.690804 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz2gd\" (UniqueName: \"kubernetes.io/projected/abdb4611-dc86-47e3-997a-afb58e26fade-kube-api-access-jz2gd\") pod \"odh-model-controller-769875db56-5ncdw\" (UID: \"abdb4611-dc86-47e3-997a-afb58e26fade\") " pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:32.690980 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.690831 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26d61838-6558-489f-a369-b8dd6afe14f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.690980 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.690867 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdb4611-dc86-47e3-997a-afb58e26fade-metrics-certs\") pod \"odh-model-controller-769875db56-5ncdw\" (UID: \"abdb4611-dc86-47e3-997a-afb58e26fade\") " pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:32.690980 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.690961 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c25kh\" (UniqueName: \"kubernetes.io/projected/26d61838-6558-489f-a369-b8dd6afe14f5-kube-api-access-c25kh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.691250 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.691006 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26d61838-6558-489f-a369-b8dd6afe14f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.691250 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.691066 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26d61838-6558-489f-a369-b8dd6afe14f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.691250 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.691203 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d61838-6558-489f-a369-b8dd6afe14f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.691250 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.691245 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26d61838-6558-489f-a369-b8dd6afe14f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.691476 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.691278 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d61838-6558-489f-a369-b8dd6afe14f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.693587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.693561 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abdb4611-dc86-47e3-997a-afb58e26fade-cert\") pod \"odh-model-controller-769875db56-5ncdw\" (UID: \"abdb4611-dc86-47e3-997a-afb58e26fade\") " pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:32.693860 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.693840 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdb4611-dc86-47e3-997a-afb58e26fade-metrics-certs\") pod \"odh-model-controller-769875db56-5ncdw\" (UID: \"abdb4611-dc86-47e3-997a-afb58e26fade\") " pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:32.705335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.705311 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz2gd\" (UniqueName: \"kubernetes.io/projected/abdb4611-dc86-47e3-997a-afb58e26fade-kube-api-access-jz2gd\") pod \"odh-model-controller-769875db56-5ncdw\" (UID: \"abdb4611-dc86-47e3-997a-afb58e26fade\") " pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:32.720212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.720184 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" Nov 02 17:45:32.757399 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.757367 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-5x726" Nov 02 17:45:32.792157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.792121 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d61838-6558-489f-a369-b8dd6afe14f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.792301 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.792238 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26d61838-6558-489f-a369-b8dd6afe14f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.792366 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.792351 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c25kh\" (UniqueName: \"kubernetes.io/projected/26d61838-6558-489f-a369-b8dd6afe14f5-kube-api-access-c25kh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.792428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.792413 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26d61838-6558-489f-a369-b8dd6afe14f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.792510 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.792491 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26d61838-6558-489f-a369-b8dd6afe14f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.792564 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.792535 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d61838-6558-489f-a369-b8dd6afe14f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.792622 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.792584 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26d61838-6558-489f-a369-b8dd6afe14f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.794004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.793929 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26d61838-6558-489f-a369-b8dd6afe14f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.795479 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.795445 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26d61838-6558-489f-a369-b8dd6afe14f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.798691 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.798664 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26d61838-6558-489f-a369-b8dd6afe14f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.798809 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.798787 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d61838-6558-489f-a369-b8dd6afe14f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.799434 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.799414 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:32.799870 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.799853 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26d61838-6558-489f-a369-b8dd6afe14f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.802139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.802118 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d61838-6558-489f-a369-b8dd6afe14f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.814042 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.814017 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c25kh\" (UniqueName: \"kubernetes.io/projected/26d61838-6558-489f-a369-b8dd6afe14f5-kube-api-access-c25kh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8f8hh\" (UID: \"26d61838-6558-489f-a369-b8dd6afe14f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.885805 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.885763 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:32.954432 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.954399 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:32.996164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:32.996122 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4pwjv"] Nov 02 17:45:32.998470 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:45:32.998430 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05242e7d_02d8_4b62_a200_cdd9f108e64a.slice/crio-15c305d02cfa2cc81d81e3fb65b93197985095cf84af1c91bd4f81d29e9ef27c WatchSource:0}: Error finding container 15c305d02cfa2cc81d81e3fb65b93197985095cf84af1c91bd4f81d29e9ef27c: Status 404 returned error can't find the container with id 15c305d02cfa2cc81d81e3fb65b93197985095cf84af1c91bd4f81d29e9ef27c Nov 02 17:45:33.043732 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:33.043700 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" event={"ID":"05242e7d-02d8-4b62-a200-cdd9f108e64a","Type":"ContainerStarted","Data":"15c305d02cfa2cc81d81e3fb65b93197985095cf84af1c91bd4f81d29e9ef27c"} Nov 02 17:45:33.049834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:33.049810 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5x726"] Nov 02 17:45:33.052489 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:45:33.052463 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94716789_2a04_4aa1_a10e_b6e2fb1517e5.slice/crio-8925cd8d4c74d76a4fd41d9fac256706f4ff7e17cb88c52027e0f365cf17f267 WatchSource:0}: Error finding container 8925cd8d4c74d76a4fd41d9fac256706f4ff7e17cb88c52027e0f365cf17f267: Status 404 returned error can't find the container with id 8925cd8d4c74d76a4fd41d9fac256706f4ff7e17cb88c52027e0f365cf17f267 Nov 02 17:45:33.099681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:33.099655 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5ncdw"] Nov 02 17:45:33.101584 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:45:33.101554 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdb4611_dc86_47e3_997a_afb58e26fade.slice/crio-fbe95157317546a64b3586e431b853c700607e1a6ac4cba762495b6e4d88d2dc WatchSource:0}: Error finding container fbe95157317546a64b3586e431b853c700607e1a6ac4cba762495b6e4d88d2dc: Status 404 returned error can't find the container with id fbe95157317546a64b3586e431b853c700607e1a6ac4cba762495b6e4d88d2dc Nov 02 17:45:33.233639 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:33.233608 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh"] Nov 02 17:45:33.235963 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:45:33.235937 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d61838_6558_489f_a369_b8dd6afe14f5.slice/crio-9f052e33eb3e73f464da75379a31e0f9535961aa2ee77a6c28590f9af5e120ce WatchSource:0}: Error finding container 9f052e33eb3e73f464da75379a31e0f9535961aa2ee77a6c28590f9af5e120ce: Status 404 returned error can't find the container with id 9f052e33eb3e73f464da75379a31e0f9535961aa2ee77a6c28590f9af5e120ce Nov 02 17:45:33.238091 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:33.238045 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:45:33.238160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:33.238129 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:45:33.525782 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:45:33.525718 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:45:33.797302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:33.797210 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:33.895824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:33.895783 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:33.994605 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:33.994567 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:34.051369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.051286 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" event={"ID":"26d61838-6558-489f-a369-b8dd6afe14f5","Type":"ContainerStarted","Data":"0f9f2ca6a7009aaa6a27d41de1861cef14e0a03dab74f17624048b1cbe12940b"} Nov 02 17:45:34.051369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.051347 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" event={"ID":"26d61838-6558-489f-a369-b8dd6afe14f5","Type":"ContainerStarted","Data":"9f052e33eb3e73f464da75379a31e0f9535961aa2ee77a6c28590f9af5e120ce"} Nov 02 17:45:34.053987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.053607 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:34.054257 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.054211 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" Nov 02 17:45:34.055641 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.055466 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-5x726" event={"ID":"94716789-2a04-4aa1-a10e-b6e2fb1517e5","Type":"ContainerStarted","Data":"8925cd8d4c74d76a4fd41d9fac256706f4ff7e17cb88c52027e0f365cf17f267"} Nov 02 17:45:34.062608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.062576 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5ncdw" event={"ID":"abdb4611-dc86-47e3-997a-afb58e26fade","Type":"ContainerStarted","Data":"fbe95157317546a64b3586e431b853c700607e1a6ac4cba762495b6e4d88d2dc"} Nov 02 17:45:34.076150 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.076092 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8f8hh" podStartSLOduration=2.076065609 podStartE2EDuration="2.076065609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:34.073861208 +0000 UTC m=+331.516943890" watchObservedRunningTime="2026-11-02 17:45:34.076065609 +0000 UTC m=+331.519148244" Nov 02 17:45:34.081160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.081060 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:34.137004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.136189 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2"] Nov 02 17:45:34.137004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.136452 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" podUID="40b795ce-68f6-4937-b1b7-c56b4c5919fe" containerName="discovery" containerID="cri-o://97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e" gracePeriod=30 Nov 02 17:45:34.514231 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.514204 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:45:34.609529 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.609474 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b795ce-68f6-4937-b1b7-c56b4c5919fe-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b795ce-68f6-4937-b1b7-c56b4c5919fe-local-certs\") pod \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " Nov 02 17:45:34.609988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.609542 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-cacerts\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-cacerts\") pod \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " Nov 02 17:45:34.609988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.609612 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-ca-configmap\") pod \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " Nov 02 17:45:34.609988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.609644 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-kubeconfig\") pod \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " Nov 02 17:45:34.609988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.609676 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-dns-cert\") pod \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " Nov 02 17:45:34.609988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.609721 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-token\" (UniqueName: \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-token\") pod \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " Nov 02 17:45:34.609988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.609781 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-kube-api-access-gnc7x\" (UniqueName: \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-kube-api-access-gnc7x\") pod \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\" (UID: \"40b795ce-68f6-4937-b1b7-c56b4c5919fe\") " Nov 02 17:45:34.610834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.610801 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-ca-configmap" pod "40b795ce-68f6-4937-b1b7-c56b4c5919fe" (UID: "40b795ce-68f6-4937-b1b7-c56b4c5919fe"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:34.624954 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.624921 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b795ce-68f6-4937-b1b7-c56b4c5919fe-local-certs" pod "40b795ce-68f6-4937-b1b7-c56b4c5919fe" (UID: "40b795ce-68f6-4937-b1b7-c56b4c5919fe"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:45:34.624954 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.624947 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-token" pod "40b795ce-68f6-4937-b1b7-c56b4c5919fe" (UID: "40b795ce-68f6-4937-b1b7-c56b4c5919fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:34.626699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.626654 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-cacerts" pod "40b795ce-68f6-4937-b1b7-c56b4c5919fe" (UID: "40b795ce-68f6-4937-b1b7-c56b4c5919fe"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:45:34.626699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.626667 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-kube-api-access-gnc7x" pod "40b795ce-68f6-4937-b1b7-c56b4c5919fe" (UID: "40b795ce-68f6-4937-b1b7-c56b4c5919fe"). InnerVolumeSpecName "kube-api-access-gnc7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:34.628353 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.628322 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-dns-cert" pod "40b795ce-68f6-4937-b1b7-c56b4c5919fe" (UID: "40b795ce-68f6-4937-b1b7-c56b4c5919fe"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:45:34.628642 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.628614 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-kubeconfig" pod "40b795ce-68f6-4937-b1b7-c56b4c5919fe" (UID: "40b795ce-68f6-4937-b1b7-c56b4c5919fe"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:45:34.711425 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.711389 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnc7x\" (UniqueName: \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-kube-api-access-gnc7x\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:34.711604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.711430 4310 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b795ce-68f6-4937-b1b7-c56b4c5919fe-local-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:34.711604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.711452 4310 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-cacerts\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:34.711604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.711467 4310 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-ca-configmap\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:34.711604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.711483 4310 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-kubeconfig\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:34.711604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.711497 4310 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-csr-dns-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:34.711604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:34.711512 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b795ce-68f6-4937-b1b7-c56b4c5919fe-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:35.076992 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:35.074535 4310 generic.go:356] "Generic (PLEG): container finished" podID="40b795ce-68f6-4937-b1b7-c56b4c5919fe" containerID="97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e" exitCode=0 Nov 02 17:45:35.076992 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:35.075432 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" Nov 02 17:45:35.076992 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:35.076112 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" event={"ID":"40b795ce-68f6-4937-b1b7-c56b4c5919fe","Type":"ContainerDied","Data":"97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e"} Nov 02 17:45:35.076992 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:35.076163 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2" event={"ID":"40b795ce-68f6-4937-b1b7-c56b4c5919fe","Type":"ContainerDied","Data":"28ee84e1321b1516a729587682de29cc818bceb5c3a55af96c318c109cd03adb"} Nov 02 17:45:35.076992 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:35.076190 4310 scope.go:122] "RemoveContainer" containerID="97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e" Nov 02 17:45:35.093258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:35.091882 4310 scope.go:122] "RemoveContainer" containerID="97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e" Nov 02 17:45:35.093258 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:45:35.092478 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e\": container with ID starting with 97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e not found: ID does not exist" containerID="97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e" Nov 02 17:45:35.093258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:35.092522 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e"} err="failed to get container status \"97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e\": rpc error: code = NotFound desc = could not find container \"97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e\": container with ID starting with 97cecd95b644d571a1fa7b6c26cf3107d7ebe0c2bee8b5e875746f8b0ac2109e not found: ID does not exist" Nov 02 17:45:35.093258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:35.093020 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2"] Nov 02 17:45:35.098042 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:35.098011 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gwgc2"] Nov 02 17:45:36.088225 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:36.088195 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" event={"ID":"05242e7d-02d8-4b62-a200-cdd9f108e64a","Type":"ContainerStarted","Data":"94e2c6f79d2c0e63bba9105047b8d33932c3e376417639df04ec28f1b6b5a183"} Nov 02 17:45:36.088574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:36.088299 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" Nov 02 17:45:36.090051 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:36.090017 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-5x726" event={"ID":"94716789-2a04-4aa1-a10e-b6e2fb1517e5","Type":"ContainerStarted","Data":"a1382cd26da86c37717f2bccaaddc0070b34e63b07469f5ea1df734de31facd9"} Nov 02 17:45:36.105901 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:36.105846 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" podStartSLOduration=1.097738571 podStartE2EDuration="4.105827896s" totalImagesPullingTime="3.008089325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:36.105170139 +0000 UTC m=+333.548252777" watchObservedRunningTime="2026-11-02 17:45:36.105827896 +0000 UTC m=+333.548910532" Nov 02 17:45:36.124289 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:36.124239 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-5x726" podStartSLOduration=1.4335701809999999 podStartE2EDuration="4.124224183s" totalImagesPullingTime="2.690654002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:36.123241137 +0000 UTC m=+333.566323774" watchObservedRunningTime="2026-11-02 17:45:36.124224183 +0000 UTC m=+333.567306819" Nov 02 17:45:36.671547 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:36.671513 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b795ce-68f6-4937-b1b7-c56b4c5919fe" path="/var/lib/kubelet/pods/40b795ce-68f6-4937-b1b7-c56b4c5919fe/volumes" Nov 02 17:45:37.067795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:37.067763 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g4jn7" Nov 02 17:45:37.096473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:37.096393 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5ncdw" event={"ID":"abdb4611-dc86-47e3-997a-afb58e26fade","Type":"ContainerStarted","Data":"308e3d645209fb5abeaf4fd511901c25a1e3f526dba83e2b62673fb5a758f8cf"} Nov 02 17:45:37.096916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:37.096521 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:37.096916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:37.096690 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-5x726" Nov 02 17:45:37.114821 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:37.114768 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-5ncdw" podStartSLOduration=1.5445172280000001 podStartE2EDuration="5.114753549s" totalImagesPullingTime="3.570236321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:37.113183718 +0000 UTC m=+334.556266365" watchObservedRunningTime="2026-11-02 17:45:37.114753549 +0000 UTC m=+334.557836178" Nov 02 17:45:43.972586 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:43.972545 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:44.036553 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.036517 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-cnlqh"] Nov 02 17:45:44.036890 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.036877 4310 cpu_manager.go:414] "containerMap: removing container" podUID="40b795ce-68f6-4937-b1b7-c56b4c5919fe" containerName="discovery" Nov 02 17:45:44.036931 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.036893 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b795ce-68f6-4937-b1b7-c56b4c5919fe" containerName="discovery" Nov 02 17:45:44.036968 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.036958 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b795ce-68f6-4937-b1b7-c56b4c5919fe" containerName="discovery" Nov 02 17:45:44.061554 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.061513 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-cnlqh"] Nov 02 17:45:44.061714 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.061641 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:44.067750 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.067723 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 17:45:44.068967 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.068940 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 17:45:44.069142 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.069045 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5xmtd\"" Nov 02 17:45:44.069243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.069126 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 17:45:44.072625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.072599 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:44.172581 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.172550 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:44.200863 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.200825 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-225cp\" (UniqueName: \"kubernetes.io/projected/36691b89-c07f-4729-8b18-ba314c5ff35b-kube-api-access-225cp\") pod \"maas-controller-579cf7c6d-cnlqh\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:44.201043 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.200879 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-webhook-cert\") pod \"maas-controller-579cf7c6d-cnlqh\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:44.201043 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.200984 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-metrics-certs\") pod \"maas-controller-579cf7c6d-cnlqh\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:44.273565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.273469 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:44.302339 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.302301 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-metrics-certs\") pod \"maas-controller-579cf7c6d-cnlqh\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:44.302508 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.302361 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-225cp\" (UniqueName: \"kubernetes.io/projected/36691b89-c07f-4729-8b18-ba314c5ff35b-kube-api-access-225cp\") pod \"maas-controller-579cf7c6d-cnlqh\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:44.302508 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.302398 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-webhook-cert\") pod \"maas-controller-579cf7c6d-cnlqh\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:44.304860 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.304830 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-metrics-certs\") pod \"maas-controller-579cf7c6d-cnlqh\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:44.304977 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.304958 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-webhook-cert\") pod \"maas-controller-579cf7c6d-cnlqh\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:44.311021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.310996 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-225cp\" (UniqueName: \"kubernetes.io/projected/36691b89-c07f-4729-8b18-ba314c5ff35b-kube-api-access-225cp\") pod \"maas-controller-579cf7c6d-cnlqh\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:44.375023 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.374985 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:44.376623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.376597 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-698c564c8b-v5qgk"] Nov 02 17:45:44.379939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.379923 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:44.393780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.393745 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-698c564c8b-v5qgk"] Nov 02 17:45:44.504633 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.504594 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twv2\" (UniqueName: \"kubernetes.io/projected/d46ea576-9209-41ed-96c0-9ced3e50d3ae-kube-api-access-8twv2\") pod \"maas-controller-698c564c8b-v5qgk\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:44.504817 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.504647 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-metrics-certs\") pod \"maas-controller-698c564c8b-v5qgk\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:44.504867 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.504838 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-webhook-cert\") pod \"maas-controller-698c564c8b-v5qgk\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:44.606057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.605964 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-webhook-cert\") pod \"maas-controller-698c564c8b-v5qgk\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:44.606057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.606039 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twv2\" (UniqueName: \"kubernetes.io/projected/d46ea576-9209-41ed-96c0-9ced3e50d3ae-kube-api-access-8twv2\") pod \"maas-controller-698c564c8b-v5qgk\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:44.606330 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.606061 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-metrics-certs\") pod \"maas-controller-698c564c8b-v5qgk\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:44.608566 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.608543 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-webhook-cert\") pod \"maas-controller-698c564c8b-v5qgk\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:44.608680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.608658 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-metrics-certs\") pod \"maas-controller-698c564c8b-v5qgk\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:44.615688 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.615661 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twv2\" (UniqueName: \"kubernetes.io/projected/d46ea576-9209-41ed-96c0-9ced3e50d3ae-kube-api-access-8twv2\") pod \"maas-controller-698c564c8b-v5qgk\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:44.660369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.660195 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-cnlqh"] Nov 02 17:45:44.662971 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:45:44.662940 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36691b89_c07f_4729_8b18_ba314c5ff35b.slice/crio-772094689fe5f409bc9b2e7925efcf5cb851d3e3734769318a237a389cae8390 WatchSource:0}: Error finding container 772094689fe5f409bc9b2e7925efcf5cb851d3e3734769318a237a389cae8390: Status 404 returned error can't find the container with id 772094689fe5f409bc9b2e7925efcf5cb851d3e3734769318a237a389cae8390 Nov 02 17:45:44.712134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.712099 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:44.985383 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:44.985287 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-698c564c8b-v5qgk"] Nov 02 17:45:44.987964 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:45:44.987937 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46ea576_9209_41ed_96c0_9ced3e50d3ae.slice/crio-1ee18a6b544a9615ebc3af783a41009b88504f59542fa95d4e0a713f6a1363eb WatchSource:0}: Error finding container 1ee18a6b544a9615ebc3af783a41009b88504f59542fa95d4e0a713f6a1363eb: Status 404 returned error can't find the container with id 1ee18a6b544a9615ebc3af783a41009b88504f59542fa95d4e0a713f6a1363eb Nov 02 17:45:45.131279 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:45.131243 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-698c564c8b-v5qgk" event={"ID":"d46ea576-9209-41ed-96c0-9ced3e50d3ae","Type":"ContainerStarted","Data":"1ee18a6b544a9615ebc3af783a41009b88504f59542fa95d4e0a713f6a1363eb"} Nov 02 17:45:45.132459 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:45.132436 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" event={"ID":"36691b89-c07f-4729-8b18-ba314c5ff35b","Type":"ContainerStarted","Data":"772094689fe5f409bc9b2e7925efcf5cb851d3e3734769318a237a389cae8390"} Nov 02 17:45:45.513711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:45.513670 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:47.906024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:47.905948 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:48.080680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:48.080645 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:48.103354 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:48.103325 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-5ncdw" Nov 02 17:45:48.105285 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:48.105260 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-5x726" Nov 02 17:45:48.149703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:48.149657 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-698c564c8b-v5qgk" event={"ID":"d46ea576-9209-41ed-96c0-9ced3e50d3ae","Type":"ContainerStarted","Data":"b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6"} Nov 02 17:45:48.149881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:48.149803 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:48.151094 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:48.151054 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" event={"ID":"36691b89-c07f-4729-8b18-ba314c5ff35b","Type":"ContainerStarted","Data":"4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a"} Nov 02 17:45:48.151236 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:48.151125 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:48.226285 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:48.226235 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" podStartSLOduration=1.244356471 podStartE2EDuration="4.226219677s" totalImagesPullingTime="2.981863206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:48.200750646 +0000 UTC m=+345.643833284" watchObservedRunningTime="2026-11-02 17:45:48.226219677 +0000 UTC m=+345.669302320" Nov 02 17:45:48.226461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:48.226442 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-698c564c8b-v5qgk" podStartSLOduration=1.4271358 podStartE2EDuration="4.226437971s" totalImagesPullingTime="2.799302171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:48.225970849 +0000 UTC m=+345.669053484" watchObservedRunningTime="2026-11-02 17:45:48.226437971 +0000 UTC m=+345.669520634" Nov 02 17:45:49.429392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.429357 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt"] Nov 02 17:45:49.433187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.433163 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.438580 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.438553 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 17:45:49.438580 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.438553 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 17:45:49.438757 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.438672 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-kvd62\"" Nov 02 17:45:49.439661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.439640 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 17:45:49.447535 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.447504 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt"] Nov 02 17:45:49.501178 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.501143 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:49.551389 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.551338 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtr7\" (UniqueName: \"kubernetes.io/projected/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-kube-api-access-cgtr7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.551569 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.551429 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.551569 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.551477 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.551569 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.551508 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.645279 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.645234 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:45:49.652790 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.652765 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.652888 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.652810 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtr7\" (UniqueName: \"kubernetes.io/projected/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-kube-api-access-cgtr7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.652888 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.652856 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.653002 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.652890 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.653474 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.653441 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.653733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.653716 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.655367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.655341 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.663648 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.663620 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtr7\" (UniqueName: \"kubernetes.io/projected/1705d3f8-7617-44ed-b8c5-6adb2e14cb55-kube-api-access-cgtr7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-8twlt\" (UID: \"1705d3f8-7617-44ed-b8c5-6adb2e14cb55\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:49.757092 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:49.757050 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:45:50.029215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:50.029192 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt"] Nov 02 17:45:50.031295 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:45:50.031263 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1705d3f8_7617_44ed_b8c5_6adb2e14cb55.slice/crio-8cb7630d7d6147f75841c1f79e080eed84de5651fde9158225ee64f03402390a WatchSource:0}: Error finding container 8cb7630d7d6147f75841c1f79e080eed84de5651fde9158225ee64f03402390a: Status 404 returned error can't find the container with id 8cb7630d7d6147f75841c1f79e080eed84de5651fde9158225ee64f03402390a Nov 02 17:45:50.161842 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:50.161808 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" event={"ID":"1705d3f8-7617-44ed-b8c5-6adb2e14cb55","Type":"ContainerStarted","Data":"8cb7630d7d6147f75841c1f79e080eed84de5651fde9158225ee64f03402390a"} Nov 02 17:45:59.158326 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.158287 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 17:45:59.158792 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.158391 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:59.236390 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.236352 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-cnlqh"] Nov 02 17:45:59.236637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.236606 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" podUID="36691b89-c07f-4729-8b18-ba314c5ff35b" containerName="manager" containerID="cri-o://4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a" gracePeriod=30 Nov 02 17:45:59.604625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.604594 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:45:59.748416 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.748380 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36691b89-c07f-4729-8b18-ba314c5ff35b-kube-api-access-225cp\" (UniqueName: \"kubernetes.io/projected/36691b89-c07f-4729-8b18-ba314c5ff35b-kube-api-access-225cp\") pod \"36691b89-c07f-4729-8b18-ba314c5ff35b\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " Nov 02 17:45:59.748615 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.748489 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-webhook-cert\") pod \"36691b89-c07f-4729-8b18-ba314c5ff35b\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " Nov 02 17:45:59.748615 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.748564 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-metrics-certs\") pod \"36691b89-c07f-4729-8b18-ba314c5ff35b\" (UID: \"36691b89-c07f-4729-8b18-ba314c5ff35b\") " Nov 02 17:45:59.751183 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.751141 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36691b89-c07f-4729-8b18-ba314c5ff35b-kube-api-access-225cp" pod "36691b89-c07f-4729-8b18-ba314c5ff35b" (UID: "36691b89-c07f-4729-8b18-ba314c5ff35b"). InnerVolumeSpecName "kube-api-access-225cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:59.751342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.751282 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-metrics-certs" pod "36691b89-c07f-4729-8b18-ba314c5ff35b" (UID: "36691b89-c07f-4729-8b18-ba314c5ff35b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:45:59.751401 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.751337 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-webhook-cert" pod "36691b89-c07f-4729-8b18-ba314c5ff35b" (UID: "36691b89-c07f-4729-8b18-ba314c5ff35b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:45:59.849319 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.849284 4310 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-metrics-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:59.849319 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.849320 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-225cp\" (UniqueName: \"kubernetes.io/projected/36691b89-c07f-4729-8b18-ba314c5ff35b-kube-api-access-225cp\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:45:59.849577 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:45:59.849333 4310 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36691b89-c07f-4729-8b18-ba314c5ff35b-webhook-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:46:00.224885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:00.224852 4310 generic.go:356] "Generic (PLEG): container finished" podID="36691b89-c07f-4729-8b18-ba314c5ff35b" containerID="4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a" exitCode=0 Nov 02 17:46:00.225329 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:00.224900 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" event={"ID":"36691b89-c07f-4729-8b18-ba314c5ff35b","Type":"ContainerDied","Data":"4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a"} Nov 02 17:46:00.225329 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:00.224916 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" Nov 02 17:46:00.225329 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:00.224943 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-cnlqh" event={"ID":"36691b89-c07f-4729-8b18-ba314c5ff35b","Type":"ContainerDied","Data":"772094689fe5f409bc9b2e7925efcf5cb851d3e3734769318a237a389cae8390"} Nov 02 17:46:00.225329 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:00.224963 4310 scope.go:122] "RemoveContainer" containerID="4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a" Nov 02 17:46:00.235231 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:00.235207 4310 scope.go:122] "RemoveContainer" containerID="4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a" Nov 02 17:46:00.235581 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:46:00.235557 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a\": container with ID starting with 4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a not found: ID does not exist" containerID="4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a" Nov 02 17:46:00.235677 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:00.235589 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a"} err="failed to get container status \"4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a\": rpc error: code = NotFound desc = could not find container \"4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a\": container with ID starting with 4c09c4c194c809a0ac249e36631dbe3854aeaea6811a858eda5b4f8a9bddea4a not found: ID does not exist" Nov 02 17:46:00.249263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:00.249227 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-cnlqh"] Nov 02 17:46:00.255143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:00.255103 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-cnlqh"] Nov 02 17:46:00.672492 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:00.672414 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36691b89-c07f-4729-8b18-ba314c5ff35b" path="/var/lib/kubelet/pods/36691b89-c07f-4729-8b18-ba314c5ff35b/volumes" Nov 02 17:46:01.231160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:01.231108 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" event={"ID":"1705d3f8-7617-44ed-b8c5-6adb2e14cb55","Type":"ContainerStarted","Data":"b9a8e1ba04546b4dcecd94115537139aed6a73cd15647ae6b81db30221e41304"} Nov 02 17:46:01.231625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:01.231183 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:46:01.252500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:01.252438 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" podStartSLOduration=1.379589406 podStartE2EDuration="12.252418831s" totalImagesPullingTime="10.872829425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:01.25004959 +0000 UTC m=+358.693132225" watchObservedRunningTime="2026-11-02 17:46:01.252418831 +0000 UTC m=+358.695501467" Nov 02 17:46:03.523107 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:46:03.523051 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:46:07.101518 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:07.101488 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4pwjv" Nov 02 17:46:12.242553 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:12.242514 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" Nov 02 17:46:17.602052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:17.602016 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:17.722652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:17.722613 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:18.891128 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:18.891062 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:19.004239 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:19.004203 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:30.135598 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:30.135564 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:30.234687 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:30.234652 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:46:33.522909 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:46:33.522871 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:46:45.556898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:46:45.556810 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:03.521932 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:47:03.521880 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:47:08.458574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:08.458537 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:08.559485 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:08.559445 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:15.266515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:15.266481 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:28.074358 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:28.074317 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:30.221451 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:30.221420 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:32.484658 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.484620 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66"] Nov 02 17:47:32.485439 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.485413 4310 cpu_manager.go:414] "containerMap: removing container" podUID="36691b89-c07f-4729-8b18-ba314c5ff35b" containerName="manager" Nov 02 17:47:32.486378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.486347 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36691b89-c07f-4729-8b18-ba314c5ff35b" containerName="manager" Nov 02 17:47:32.486493 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.486484 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="36691b89-c07f-4729-8b18-ba314c5ff35b" containerName="manager" Nov 02 17:47:32.489887 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.489869 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.494269 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.494236 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-59qm5\"" Nov 02 17:47:32.494378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.494273 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 17:47:32.494378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.494330 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 17:47:32.494378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.494347 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 17:47:32.499573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.499539 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66"] Nov 02 17:47:32.545068 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.545035 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:32.595298 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.595263 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.595482 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.595343 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.595482 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.595373 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2144453b-de91-4404-8b54-5960b5a697ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.595482 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.595398 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.595614 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.595499 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drf8t\" (UniqueName: \"kubernetes.io/projected/2144453b-de91-4404-8b54-5960b5a697ce-kube-api-access-drf8t\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.595614 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.595542 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.696323 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.696281 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.696323 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.696315 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2144453b-de91-4404-8b54-5960b5a697ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.696600 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.696423 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.696600 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.696468 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drf8t\" (UniqueName: \"kubernetes.io/projected/2144453b-de91-4404-8b54-5960b5a697ce-kube-api-access-drf8t\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.696600 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.696505 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.696600 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.696555 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.696827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.696808 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.696878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.696858 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.697003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.696981 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.698561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.698543 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2144453b-de91-4404-8b54-5960b5a697ce-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.698868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.698852 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2144453b-de91-4404-8b54-5960b5a697ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.705235 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.705211 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drf8t\" (UniqueName: \"kubernetes.io/projected/2144453b-de91-4404-8b54-5960b5a697ce-kube-api-access-drf8t\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jdv66\" (UID: \"2144453b-de91-4404-8b54-5960b5a697ce\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:32.803188 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:32.803061 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:33.086826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.086799 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66"] Nov 02 17:47:33.088702 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:47:33.088673 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2144453b_de91_4404_8b54_5960b5a697ce.slice/crio-5e820225487da9cad5ceae5a3ffd36d0af1cb3630a637325412bc34cfb67c6f5 WatchSource:0}: Error finding container 5e820225487da9cad5ceae5a3ffd36d0af1cb3630a637325412bc34cfb67c6f5: Status 404 returned error can't find the container with id 5e820225487da9cad5ceae5a3ffd36d0af1cb3630a637325412bc34cfb67c6f5 Nov 02 17:47:33.527838 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:47:33.527799 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:47:33.601151 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.601113 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg"] Nov 02 17:47:33.605181 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.605156 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.611265 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.611230 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" event={"ID":"2144453b-de91-4404-8b54-5960b5a697ce","Type":"ContainerStarted","Data":"5e820225487da9cad5ceae5a3ffd36d0af1cb3630a637325412bc34cfb67c6f5"} Nov 02 17:47:33.613239 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.613203 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 17:47:33.617581 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.617557 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg"] Nov 02 17:47:33.666696 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.666632 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:33.705991 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.705953 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b41c5ba-f619-4894-8e4f-427014637518-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.706200 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.706013 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.706200 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.706155 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.706350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.706242 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78wbg\" (UniqueName: \"kubernetes.io/projected/3b41c5ba-f619-4894-8e4f-427014637518-kube-api-access-78wbg\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.706350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.706332 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.706518 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.706388 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.807216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.807128 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78wbg\" (UniqueName: \"kubernetes.io/projected/3b41c5ba-f619-4894-8e4f-427014637518-kube-api-access-78wbg\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.807216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.807198 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.807414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.807236 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.807414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.807265 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b41c5ba-f619-4894-8e4f-427014637518-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.807414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.807305 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.807414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.807370 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.808165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.808129 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.808296 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.808173 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.808529 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.808505 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.810489 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.810469 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b41c5ba-f619-4894-8e4f-427014637518-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.810588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.810500 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b41c5ba-f619-4894-8e4f-427014637518-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.816608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.816586 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78wbg\" (UniqueName: \"kubernetes.io/projected/3b41c5ba-f619-4894-8e4f-427014637518-kube-api-access-78wbg\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4v5sg\" (UID: \"3b41c5ba-f619-4894-8e4f-427014637518\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:33.924693 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:33.924657 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:34.209829 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:34.209691 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg"] Nov 02 17:47:34.212214 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:47:34.212172 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b41c5ba_f619_4894_8e4f_427014637518.slice/crio-c02d9bbc168d554a44636c769a44409db9f4d796b666a553b5b167e565fd2c6f WatchSource:0}: Error finding container c02d9bbc168d554a44636c769a44409db9f4d796b666a553b5b167e565fd2c6f: Status 404 returned error can't find the container with id c02d9bbc168d554a44636c769a44409db9f4d796b666a553b5b167e565fd2c6f Nov 02 17:47:34.618114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:34.618060 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" event={"ID":"3b41c5ba-f619-4894-8e4f-427014637518","Type":"ContainerStarted","Data":"0a3597d2f7644019d096c2fba6420b6aa30655e7bffd15869e3352a26e21566a"} Nov 02 17:47:34.618590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:34.618147 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" event={"ID":"3b41c5ba-f619-4894-8e4f-427014637518","Type":"ContainerStarted","Data":"c02d9bbc168d554a44636c769a44409db9f4d796b666a553b5b167e565fd2c6f"} Nov 02 17:47:34.619643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:34.619618 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" event={"ID":"2144453b-de91-4404-8b54-5960b5a697ce","Type":"ContainerStarted","Data":"9e74675112079dd7dc20d0f7988cd70cf49922d82c4c53f693825ac2f95ece48"} Nov 02 17:47:34.635292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:34.635244 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" podStartSLOduration=1.314246773 podStartE2EDuration="1.635231175s" totalImagesPullingTime="320.984402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:34.634006022 +0000 UTC m=+452.077088658" watchObservedRunningTime="2026-11-02 17:47:34.635231175 +0000 UTC m=+452.078313829" Nov 02 17:47:34.652690 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:34.652628 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" podStartSLOduration=1.33993954 podStartE2EDuration="2.65260879s" totalImagesPullingTime="1.31266925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:34.650984942 +0000 UTC m=+452.094067579" watchObservedRunningTime="2026-11-02 17:47:34.65260879 +0000 UTC m=+452.095691427" Nov 02 17:47:34.803358 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:34.803281 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:34.925139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:34.925101 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:35.117581 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:35.117501 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:36.109537 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.109501 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp"] Nov 02 17:47:36.113220 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.113200 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.116749 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.116727 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 17:47:36.127063 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.127029 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp"] Nov 02 17:47:36.184760 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.184732 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:36.228073 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.228033 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.228308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.228105 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.228308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.228165 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4955081-1a71-4933-8332-dd8bc035ae18-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.228308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.228204 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.228308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.228233 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.228308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.228285 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qnfl\" (UniqueName: \"kubernetes.io/projected/d4955081-1a71-4933-8332-dd8bc035ae18-kube-api-access-7qnfl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.329472 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.329437 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.329650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.329483 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.329650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.329535 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4955081-1a71-4933-8332-dd8bc035ae18-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.329650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.329585 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.329650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.329611 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.329854 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.329663 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qnfl\" (UniqueName: \"kubernetes.io/projected/d4955081-1a71-4933-8332-dd8bc035ae18-kube-api-access-7qnfl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.329946 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.329926 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.330072 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.330049 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.330171 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.330073 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.332406 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.332388 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4955081-1a71-4933-8332-dd8bc035ae18-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.332806 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.332785 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4955081-1a71-4933-8332-dd8bc035ae18-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.343884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.343859 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qnfl\" (UniqueName: \"kubernetes.io/projected/d4955081-1a71-4933-8332-dd8bc035ae18-kube-api-access-7qnfl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp\" (UID: \"d4955081-1a71-4933-8332-dd8bc035ae18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.427048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.426963 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:36.711176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:36.711142 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp"] Nov 02 17:47:36.713709 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:47:36.713676 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4955081_1a71_4933_8332_dd8bc035ae18.slice/crio-f96a359fadc0daa53a357fc6a26e0ddc6c7b820b016ae6479f1a1004375f8087 WatchSource:0}: Error finding container f96a359fadc0daa53a357fc6a26e0ddc6c7b820b016ae6479f1a1004375f8087: Status 404 returned error can't find the container with id f96a359fadc0daa53a357fc6a26e0ddc6c7b820b016ae6479f1a1004375f8087 Nov 02 17:47:37.133039 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.133007 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6jcls"] Nov 02 17:47:37.136532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.136506 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.143331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.143288 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 17:47:37.150254 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.150225 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6jcls"] Nov 02 17:47:37.185356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.185323 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:37.237939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.237900 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hxfr\" (UniqueName: \"kubernetes.io/projected/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-kube-api-access-5hxfr\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.238141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.237961 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-home\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.238141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.238045 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.238141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.238095 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.238268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.238195 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.238268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.238226 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.339578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.339498 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-home\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.339578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.339547 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.339782 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.339663 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.339782 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.339704 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.339891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.339810 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.339891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.339861 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hxfr\" (UniqueName: \"kubernetes.io/projected/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-kube-api-access-5hxfr\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.339891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.339879 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-home\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.340046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.340011 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.340140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.340124 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.341899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.341877 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.341993 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.341901 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.347536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.347516 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hxfr\" (UniqueName: \"kubernetes.io/projected/c19cdb66-3f5e-4ac8-a77c-d0a85d130613-kube-api-access-5hxfr\") pod \"echo-simulator-kserve-64b6f8f96c-6jcls\" (UID: \"c19cdb66-3f5e-4ac8-a77c-d0a85d130613\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.452884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.452841 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:37.633823 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.633738 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" event={"ID":"d4955081-1a71-4933-8332-dd8bc035ae18","Type":"ContainerStarted","Data":"aea38f878ee1d02385a68dabdc131262ab38966c5d4f42129ac12cb9749a249f"} Nov 02 17:47:37.633823 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.633791 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" event={"ID":"d4955081-1a71-4933-8332-dd8bc035ae18","Type":"ContainerStarted","Data":"f96a359fadc0daa53a357fc6a26e0ddc6c7b820b016ae6479f1a1004375f8087"} Nov 02 17:47:37.651873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.651821 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" podStartSLOduration=1.328077176 podStartE2EDuration="1.651806495s" totalImagesPullingTime="323.729319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:37.650433532 +0000 UTC m=+455.093516179" watchObservedRunningTime="2026-11-02 17:47:37.651806495 +0000 UTC m=+455.094889131" Nov 02 17:47:37.723864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:37.723839 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6jcls"] Nov 02 17:47:37.726432 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:47:37.726406 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19cdb66_3f5e_4ac8_a77c_d0a85d130613.slice/crio-d04628e24851418a97658ba03a423c999dfa95369c2b57d2dadecbadc65b6df0 WatchSource:0}: Error finding container d04628e24851418a97658ba03a423c999dfa95369c2b57d2dadecbadc65b6df0: Status 404 returned error can't find the container with id d04628e24851418a97658ba03a423c999dfa95369c2b57d2dadecbadc65b6df0 Nov 02 17:47:38.427980 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:38.427745 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:38.639658 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:38.639611 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" event={"ID":"c19cdb66-3f5e-4ac8-a77c-d0a85d130613","Type":"ContainerStarted","Data":"d04628e24851418a97658ba03a423c999dfa95369c2b57d2dadecbadc65b6df0"} Nov 02 17:47:38.771959 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:38.771904 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:39.646533 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:39.646494 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" event={"ID":"c19cdb66-3f5e-4ac8-a77c-d0a85d130613","Type":"ContainerStarted","Data":"58b4934a4ab63ac10920c3f1bb57356c268f28151eb6b32f4987370293081e2a"} Nov 02 17:47:39.665061 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:39.665014 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" podStartSLOduration=0.838204027 podStartE2EDuration="2.66499985s" totalImagesPullingTime="1.826795823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:39.664177509 +0000 UTC m=+457.107260145" watchObservedRunningTime="2026-11-02 17:47:39.66499985 +0000 UTC m=+457.108082535" Nov 02 17:47:40.453651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:40.453613 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:42.803760 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:42.803721 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:42.816470 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:42.816439 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:43.674281 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:43.674245 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" Nov 02 17:47:43.925398 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:43.925291 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:43.938054 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:43.938021 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:44.683196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:44.683158 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" Nov 02 17:47:45.596298 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:45.596261 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:46.427492 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:46.427450 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:46.440018 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:46.439981 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:46.688945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:46.688867 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" Nov 02 17:47:47.453349 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:47.453313 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:47.465727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:47.465695 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:47.693188 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:47.693158 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:47:54.889689 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:54.889645 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:54.909708 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:54.909674 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-2vkbv"] Nov 02 17:47:54.913118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:54.913103 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:47:54.918720 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:54.918698 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 17:47:54.918845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:54.918718 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 17:47:54.918845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:54.918725 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-cdk6l\"" Nov 02 17:47:54.922913 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:54.922888 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-2vkbv"] Nov 02 17:47:54.985237 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:54.985198 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:47:55.003578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:55.003539 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfxqq\" (UniqueName: \"kubernetes.io/projected/10db5c18-8e82-4506-b11b-792d3bb3b639-kube-api-access-sfxqq\") pod \"ai-gateway-controller-b59d66cb5-2vkbv\" (UID: \"10db5c18-8e82-4506-b11b-792d3bb3b639\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:47:55.003759 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:55.003675 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10db5c18-8e82-4506-b11b-792d3bb3b639-metrics-certs\") pod \"ai-gateway-controller-b59d66cb5-2vkbv\" (UID: \"10db5c18-8e82-4506-b11b-792d3bb3b639\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:47:55.104506 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:55.104468 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfxqq\" (UniqueName: \"kubernetes.io/projected/10db5c18-8e82-4506-b11b-792d3bb3b639-kube-api-access-sfxqq\") pod \"ai-gateway-controller-b59d66cb5-2vkbv\" (UID: \"10db5c18-8e82-4506-b11b-792d3bb3b639\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:47:55.104707 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:55.104634 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10db5c18-8e82-4506-b11b-792d3bb3b639-metrics-certs\") pod \"ai-gateway-controller-b59d66cb5-2vkbv\" (UID: \"10db5c18-8e82-4506-b11b-792d3bb3b639\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:47:55.107063 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:55.107035 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10db5c18-8e82-4506-b11b-792d3bb3b639-metrics-certs\") pod \"ai-gateway-controller-b59d66cb5-2vkbv\" (UID: \"10db5c18-8e82-4506-b11b-792d3bb3b639\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:47:55.112498 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:55.112480 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfxqq\" (UniqueName: \"kubernetes.io/projected/10db5c18-8e82-4506-b11b-792d3bb3b639-kube-api-access-sfxqq\") pod \"ai-gateway-controller-b59d66cb5-2vkbv\" (UID: \"10db5c18-8e82-4506-b11b-792d3bb3b639\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:47:55.227544 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:55.227512 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:47:55.499358 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:55.499331 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-2vkbv"] Nov 02 17:47:55.501972 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:47:55.501938 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10db5c18_8e82_4506_b11b_792d3bb3b639.slice/crio-d1410a1022309ae2a61cc311ca2f351c095b1abbdc58f6e580be39febd4515dd WatchSource:0}: Error finding container d1410a1022309ae2a61cc311ca2f351c095b1abbdc58f6e580be39febd4515dd: Status 404 returned error can't find the container with id d1410a1022309ae2a61cc311ca2f351c095b1abbdc58f6e580be39febd4515dd Nov 02 17:47:55.720747 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:55.720713 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" event={"ID":"10db5c18-8e82-4506-b11b-792d3bb3b639","Type":"ContainerStarted","Data":"d1410a1022309ae2a61cc311ca2f351c095b1abbdc58f6e580be39febd4515dd"} Nov 02 17:47:59.740258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:59.740220 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" event={"ID":"10db5c18-8e82-4506-b11b-792d3bb3b639","Type":"ContainerStarted","Data":"8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023"} Nov 02 17:47:59.740651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:59.740319 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:47:59.759425 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:47:59.759365 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" podStartSLOduration=1.7517117 podStartE2EDuration="5.759345262s" totalImagesPullingTime="4.007633562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:59.758223211 +0000 UTC m=+477.201305847" watchObservedRunningTime="2026-11-02 17:47:59.759345262 +0000 UTC m=+477.202427899" Nov 02 17:48:01.310616 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.310580 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:01.336908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.336845 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh"] Nov 02 17:48:01.341168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.341142 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.351115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.350976 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zplmb\"" Nov 02 17:48:01.357609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.357576 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh"] Nov 02 17:48:01.443908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.443863 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:01.466013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.465978 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a783ce1-3916-4723-a400-9155ea747d15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.466193 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.466016 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a783ce1-3916-4723-a400-9155ea747d15-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.466193 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.466120 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.466193 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.466156 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.466193 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.466179 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a783ce1-3916-4723-a400-9155ea747d15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.466325 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.466224 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqsnl\" (UniqueName: \"kubernetes.io/projected/1a783ce1-3916-4723-a400-9155ea747d15-kube-api-access-lqsnl\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.466325 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.466300 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.466387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.466335 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.466461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.466449 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.567879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.567783 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqsnl\" (UniqueName: \"kubernetes.io/projected/1a783ce1-3916-4723-a400-9155ea747d15-kube-api-access-lqsnl\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.567879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.567828 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.567937 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.568040 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.568107 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a783ce1-3916-4723-a400-9155ea747d15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.568144 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a783ce1-3916-4723-a400-9155ea747d15-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.568163 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.568182 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.568200 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a783ce1-3916-4723-a400-9155ea747d15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.568255 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.568305 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.568451 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.568595 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.568883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.568863 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a783ce1-3916-4723-a400-9155ea747d15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.570716 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.570686 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a783ce1-3916-4723-a400-9155ea747d15-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.570854 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.570728 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a783ce1-3916-4723-a400-9155ea747d15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.575727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.575700 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a783ce1-3916-4723-a400-9155ea747d15-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.575829 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.575780 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqsnl\" (UniqueName: \"kubernetes.io/projected/1a783ce1-3916-4723-a400-9155ea747d15-kube-api-access-lqsnl\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkwgh\" (UID: \"1a783ce1-3916-4723-a400-9155ea747d15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.655454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.655420 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:01.938614 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:01.938586 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh"] Nov 02 17:48:01.940892 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:48:01.940854 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a783ce1_3916_4723_a400_9155ea747d15.slice/crio-e888e27ca737fcb5f784060d13efcd25bc123370f16ceb31c5984b6fb4ad72f8 WatchSource:0}: Error finding container e888e27ca737fcb5f784060d13efcd25bc123370f16ceb31c5984b6fb4ad72f8: Status 404 returned error can't find the container with id e888e27ca737fcb5f784060d13efcd25bc123370f16ceb31c5984b6fb4ad72f8 Nov 02 17:48:02.754868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:02.754832 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" event={"ID":"1a783ce1-3916-4723-a400-9155ea747d15","Type":"ContainerStarted","Data":"e888e27ca737fcb5f784060d13efcd25bc123370f16ceb31c5984b6fb4ad72f8"} Nov 02 17:48:03.522023 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:48:03.521984 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:48:10.746722 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:10.746690 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:48:11.020396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.020318 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-fdd7ff9-r2xj6"] Nov 02 17:48:11.028505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.028478 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" Nov 02 17:48:11.036054 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.036026 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-fdd7ff9-r2xj6"] Nov 02 17:48:11.159530 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.159491 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325e80f9-6751-44b2-a3b2-0c2b1d2e4def-metrics-certs\") pod \"ai-gateway-controller-fdd7ff9-r2xj6\" (UID: \"325e80f9-6751-44b2-a3b2-0c2b1d2e4def\") " pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" Nov 02 17:48:11.159530 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.159535 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tkpg\" (UniqueName: \"kubernetes.io/projected/325e80f9-6751-44b2-a3b2-0c2b1d2e4def-kube-api-access-6tkpg\") pod \"ai-gateway-controller-fdd7ff9-r2xj6\" (UID: \"325e80f9-6751-44b2-a3b2-0c2b1d2e4def\") " pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" Nov 02 17:48:11.260719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.260677 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325e80f9-6751-44b2-a3b2-0c2b1d2e4def-metrics-certs\") pod \"ai-gateway-controller-fdd7ff9-r2xj6\" (UID: \"325e80f9-6751-44b2-a3b2-0c2b1d2e4def\") " pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" Nov 02 17:48:11.260719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.260718 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tkpg\" (UniqueName: \"kubernetes.io/projected/325e80f9-6751-44b2-a3b2-0c2b1d2e4def-kube-api-access-6tkpg\") pod \"ai-gateway-controller-fdd7ff9-r2xj6\" (UID: \"325e80f9-6751-44b2-a3b2-0c2b1d2e4def\") " pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" Nov 02 17:48:11.263149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.263125 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325e80f9-6751-44b2-a3b2-0c2b1d2e4def-metrics-certs\") pod \"ai-gateway-controller-fdd7ff9-r2xj6\" (UID: \"325e80f9-6751-44b2-a3b2-0c2b1d2e4def\") " pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" Nov 02 17:48:11.270709 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.270649 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tkpg\" (UniqueName: \"kubernetes.io/projected/325e80f9-6751-44b2-a3b2-0c2b1d2e4def-kube-api-access-6tkpg\") pod \"ai-gateway-controller-fdd7ff9-r2xj6\" (UID: \"325e80f9-6751-44b2-a3b2-0c2b1d2e4def\") " pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" Nov 02 17:48:11.343200 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.343157 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" Nov 02 17:48:11.615709 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.615683 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-fdd7ff9-r2xj6"] Nov 02 17:48:11.617752 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:48:11.617718 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325e80f9_6751_44b2_a3b2_0c2b1d2e4def.slice/crio-a2c4894e3bb5cb5b0c223d9d17813a14a09a06389a24260d9f3cf7f5cc07be97 WatchSource:0}: Error finding container a2c4894e3bb5cb5b0c223d9d17813a14a09a06389a24260d9f3cf7f5cc07be97: Status 404 returned error can't find the container with id a2c4894e3bb5cb5b0c223d9d17813a14a09a06389a24260d9f3cf7f5cc07be97 Nov 02 17:48:11.793950 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:11.793912 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" event={"ID":"325e80f9-6751-44b2-a3b2-0c2b1d2e4def","Type":"ContainerStarted","Data":"a2c4894e3bb5cb5b0c223d9d17813a14a09a06389a24260d9f3cf7f5cc07be97"} Nov 02 17:48:12.799327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:12.799291 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" event={"ID":"325e80f9-6751-44b2-a3b2-0c2b1d2e4def","Type":"ContainerStarted","Data":"fb8eacd9f5fd01d8f928575de2f2b1994f4c68a4aa7e20ec95cebccb4c023186"} Nov 02 17:48:12.799704 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:12.799366 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" Nov 02 17:48:12.817466 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:12.817420 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" podStartSLOduration=1.3790897819999999 podStartE2EDuration="1.817405441s" totalImagesPullingTime="438.315659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:12.816317536 +0000 UTC m=+490.259400173" watchObservedRunningTime="2026-11-02 17:48:12.817405441 +0000 UTC m=+490.260488077" Nov 02 17:48:16.374298 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.374220 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd"] Nov 02 17:48:16.377817 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.377796 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.393365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.393338 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd"] Nov 02 17:48:16.395707 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.395681 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.395830 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.395717 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f41e4-d435-4f1f-95dc-858820a331bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.395830 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.395789 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.395830 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.395818 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcqr\" (UniqueName: \"kubernetes.io/projected/8d9f41e4-d435-4f1f-95dc-858820a331bf-kube-api-access-zhcqr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.395938 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.395841 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.395938 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.395859 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.396014 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.395924 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.396014 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.395981 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.396109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.396044 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.496620 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.496580 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.496620 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.496625 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcqr\" (UniqueName: \"kubernetes.io/projected/8d9f41e4-d435-4f1f-95dc-858820a331bf-kube-api-access-zhcqr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.496875 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.496740 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.496875 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.496777 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.496875 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.496810 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.496875 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.496852 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.497189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.496882 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.497189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.496997 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.497189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.497030 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f41e4-d435-4f1f-95dc-858820a331bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.497189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.497069 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.497491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.497243 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.497491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.497402 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.497491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.497454 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.497757 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.497735 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f41e4-d435-4f1f-95dc-858820a331bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.499320 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.499302 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.499637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.499616 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.505430 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.505397 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcqr\" (UniqueName: \"kubernetes.io/projected/8d9f41e4-d435-4f1f-95dc-858820a331bf-kube-api-access-zhcqr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.505948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.505921 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9f41e4-d435-4f1f-95dc-858820a331bf-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8ftbd\" (UID: \"8d9f41e4-d435-4f1f-95dc-858820a331bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.691968 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.691892 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:16.972509 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:16.972483 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd"] Nov 02 17:48:16.975166 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:48:16.975139 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9f41e4_d435_4f1f_95dc_858820a331bf.slice/crio-1d218e47986d9793a399456700a267ad8a89f5877bcf8e099ef022283e887936 WatchSource:0}: Error finding container 1d218e47986d9793a399456700a267ad8a89f5877bcf8e099ef022283e887936: Status 404 returned error can't find the container with id 1d218e47986d9793a399456700a267ad8a89f5877bcf8e099ef022283e887936 Nov 02 17:48:17.822351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:17.822316 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" event={"ID":"8d9f41e4-d435-4f1f-95dc-858820a331bf","Type":"ContainerStarted","Data":"1d218e47986d9793a399456700a267ad8a89f5877bcf8e099ef022283e887936"} Nov 02 17:48:20.069623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:20.069593 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:48:20.069962 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:20.069666 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:48:20.069962 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:20.069701 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:48:20.578749 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:20.578717 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:48:20.578835 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:20.578791 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:48:20.578835 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:20.578821 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:48:20.842344 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:20.842247 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" event={"ID":"1a783ce1-3916-4723-a400-9155ea747d15","Type":"ContainerStarted","Data":"b0e0ba47553e92bb6897a36c7738a283620f735bef88f2703374934781c511b1"} Nov 02 17:48:20.861100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:20.860732 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" event={"ID":"8d9f41e4-d435-4f1f-95dc-858820a331bf","Type":"ContainerStarted","Data":"28711a9b3f37c8982e927e588fd6210dfe937049de03d0a529d3c4c8dbdcd89b"} Nov 02 17:48:20.876595 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:20.876514 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" podStartSLOduration=1.241979486 podStartE2EDuration="19.876498106s" totalImagesPullingTime="18.63451862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:20.876367058 +0000 UTC m=+498.319449695" watchObservedRunningTime="2026-11-02 17:48:20.876498106 +0000 UTC m=+498.319580742" Nov 02 17:48:20.902004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:20.901953 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" podStartSLOduration=1.810764169 podStartE2EDuration="4.901938541s" totalImagesPullingTime="3.091174372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:20.900911833 +0000 UTC m=+498.343994494" watchObservedRunningTime="2026-11-02 17:48:20.901938541 +0000 UTC m=+498.345021177" Nov 02 17:48:21.672102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:21.668598 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:21.672102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:21.670528 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkwgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:48:21.672102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:21.670581 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" podUID="1a783ce1-3916-4723-a400-9155ea747d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:48:21.693266 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:21.692709 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:21.694872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:21.694844 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8ftbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:48:21.696540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:21.696495 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" podUID="8d9f41e4-d435-4f1f-95dc-858820a331bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:48:22.658681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:22.658646 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkwgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:48:22.658969 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:22.658937 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" podUID="1a783ce1-3916-4723-a400-9155ea747d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:48:22.695623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:22.695587 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8ftbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:48:22.697241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:22.696154 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" podUID="8d9f41e4-d435-4f1f-95dc-858820a331bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:48:23.658578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:23.658538 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkwgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:48:23.658865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:23.658838 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" podUID="1a783ce1-3916-4723-a400-9155ea747d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:48:23.695583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:23.695546 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8ftbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:48:23.696286 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:23.695854 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" podUID="8d9f41e4-d435-4f1f-95dc-858820a331bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:48:23.806258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:23.806232 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-fdd7ff9-r2xj6" Nov 02 17:48:23.894249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:23.894197 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-2vkbv"] Nov 02 17:48:23.895021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:23.894990 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" podUID="10db5c18-8e82-4506-b11b-792d3bb3b639" containerName="manager" containerID="cri-o://8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023" gracePeriod=30 Nov 02 17:48:24.262118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.261402 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:48:24.312633 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.312598 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10db5c18-8e82-4506-b11b-792d3bb3b639-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10db5c18-8e82-4506-b11b-792d3bb3b639-metrics-certs\") pod \"10db5c18-8e82-4506-b11b-792d3bb3b639\" (UID: \"10db5c18-8e82-4506-b11b-792d3bb3b639\") " Nov 02 17:48:24.312805 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.312672 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10db5c18-8e82-4506-b11b-792d3bb3b639-kube-api-access-sfxqq\" (UniqueName: \"kubernetes.io/projected/10db5c18-8e82-4506-b11b-792d3bb3b639-kube-api-access-sfxqq\") pod \"10db5c18-8e82-4506-b11b-792d3bb3b639\" (UID: \"10db5c18-8e82-4506-b11b-792d3bb3b639\") " Nov 02 17:48:24.314775 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.314746 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10db5c18-8e82-4506-b11b-792d3bb3b639-metrics-certs" pod "10db5c18-8e82-4506-b11b-792d3bb3b639" (UID: "10db5c18-8e82-4506-b11b-792d3bb3b639"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:24.314885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.314778 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10db5c18-8e82-4506-b11b-792d3bb3b639-kube-api-access-sfxqq" pod "10db5c18-8e82-4506-b11b-792d3bb3b639" (UID: "10db5c18-8e82-4506-b11b-792d3bb3b639"). InnerVolumeSpecName "kube-api-access-sfxqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:24.414216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.414179 4310 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10db5c18-8e82-4506-b11b-792d3bb3b639-metrics-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:24.414216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.414211 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfxqq\" (UniqueName: \"kubernetes.io/projected/10db5c18-8e82-4506-b11b-792d3bb3b639-kube-api-access-sfxqq\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:48:24.659636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.659536 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkwgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:48:24.659901 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.659869 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" podUID="1a783ce1-3916-4723-a400-9155ea747d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:48:24.696138 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.693483 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8ftbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:48:24.696138 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.693555 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" podUID="8d9f41e4-d435-4f1f-95dc-858820a331bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:48:24.908256 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.908218 4310 generic.go:356] "Generic (PLEG): container finished" podID="10db5c18-8e82-4506-b11b-792d3bb3b639" containerID="8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023" exitCode=0 Nov 02 17:48:24.908421 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.908272 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" Nov 02 17:48:24.908421 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.908279 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" event={"ID":"10db5c18-8e82-4506-b11b-792d3bb3b639","Type":"ContainerDied","Data":"8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023"} Nov 02 17:48:24.908421 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.908334 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b59d66cb5-2vkbv" event={"ID":"10db5c18-8e82-4506-b11b-792d3bb3b639","Type":"ContainerDied","Data":"d1410a1022309ae2a61cc311ca2f351c095b1abbdc58f6e580be39febd4515dd"} Nov 02 17:48:24.908421 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.908355 4310 scope.go:122] "RemoveContainer" containerID="8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023" Nov 02 17:48:24.916863 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.916845 4310 scope.go:122] "RemoveContainer" containerID="8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023" Nov 02 17:48:24.917135 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:48:24.917115 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023\": container with ID starting with 8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023 not found: ID does not exist" containerID="8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023" Nov 02 17:48:24.917203 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.917145 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023"} err="failed to get container status \"8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023\": rpc error: code = NotFound desc = could not find container \"8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023\": container with ID starting with 8caae9722dd8951ca88753e94b75ac9849881b6ffe14fd709cca3a797c614023 not found: ID does not exist" Nov 02 17:48:24.927808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.927786 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-2vkbv"] Nov 02 17:48:24.932119 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:24.932094 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-2vkbv"] Nov 02 17:48:25.499632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:25.499587 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:25.659115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:25.658643 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkwgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:48:25.659115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:25.658706 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" podUID="1a783ce1-3916-4723-a400-9155ea747d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:48:25.698734 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:25.698700 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8ftbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:48:25.699374 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:25.699334 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" podUID="8d9f41e4-d435-4f1f-95dc-858820a331bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:48:26.656387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:26.656347 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkwgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:48:26.656565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:26.656409 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" podUID="1a783ce1-3916-4723-a400-9155ea747d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:48:26.673200 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:26.673170 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10db5c18-8e82-4506-b11b-792d3bb3b639" path="/var/lib/kubelet/pods/10db5c18-8e82-4506-b11b-792d3bb3b639/volumes" Nov 02 17:48:26.693174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:26.693132 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8ftbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:48:26.693331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:26.693195 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" podUID="8d9f41e4-d435-4f1f-95dc-858820a331bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:48:26.694431 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:26.694387 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:27.480383 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:27.480337 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:27.591173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:27.591131 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:27.656032 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:27.655994 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkwgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:48:27.656212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:27.656049 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" podUID="1a783ce1-3916-4723-a400-9155ea747d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 17:48:27.692574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:27.692531 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8ftbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:48:27.692748 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:27.692595 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" podUID="8d9f41e4-d435-4f1f-95dc-858820a331bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 17:48:28.696747 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:28.696715 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:28.697247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:28.697065 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:28.697879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:28.697859 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" Nov 02 17:48:28.955381 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:28.955282 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:28.975288 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:28.975204 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:28.975965 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:28.975637 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:28.976990 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:28.976961 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" Nov 02 17:48:29.103292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:29.103259 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:33.520864 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:48:33.520824 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:48:34.089288 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:34.089252 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:39.570979 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:39.570940 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:48:45.653124 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:48:45.653089 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:03.522370 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:49:03.522330 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:49:06.174891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.174849 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:06.292823 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.292788 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:06.650741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.650691 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:06.659740 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.659710 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k"] Nov 02 17:49:06.660157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.660142 4310 cpu_manager.go:414] "containerMap: removing container" podUID="10db5c18-8e82-4506-b11b-792d3bb3b639" containerName="manager" Nov 02 17:49:06.660209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.660161 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10db5c18-8e82-4506-b11b-792d3bb3b639" containerName="manager" Nov 02 17:49:06.660243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.660234 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="10db5c18-8e82-4506-b11b-792d3bb3b639" containerName="manager" Nov 02 17:49:06.667812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.667789 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.680741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.680715 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ea752f-openshift-default-dockercfg-dsvkg\"" Nov 02 17:49:06.683071 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.683039 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k"] Nov 02 17:49:06.706490 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.706454 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.706490 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.706487 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.706718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.706515 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4svw9\" (UniqueName: \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-kube-api-access-4svw9\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.706718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.706620 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.706718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.706645 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.706718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.706699 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d182005-2390-4c46-8327-c051ce27933a-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.706718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.706720 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.706990 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.706804 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d182005-2390-4c46-8327-c051ce27933a-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.706990 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.706834 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.780204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.780171 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng"] Nov 02 17:49:06.783796 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.783773 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.790454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.790430 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9dfa27-openshift-default-dockercfg-z57p2\"" Nov 02 17:49:06.801208 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.801177 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng"] Nov 02 17:49:06.807781 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.807758 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.807908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.807786 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.807908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.807807 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d182005-2390-4c46-8327-c051ce27933a-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.807908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.807857 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.807908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.807900 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.808117 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.807976 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.808117 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808037 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d182005-2390-4c46-8327-c051ce27933a-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.808117 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808062 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.808272 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808204 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.808272 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808240 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.808371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808265 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.808371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808283 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.808371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808312 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.808371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808341 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.808576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808374 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.808576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808440 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w447\" (UniqueName: \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-kube-api-access-7w447\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.808576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808529 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.808576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808567 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.808776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808524 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.808776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808599 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.808776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808633 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svw9\" (UniqueName: \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-kube-api-access-4svw9\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.808938 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.808857 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.809395 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.809374 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d182005-2390-4c46-8327-c051ce27933a-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.810423 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.810405 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d182005-2390-4c46-8327-c051ce27933a-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.810518 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.810448 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.825815 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.825789 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.833941 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.833913 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svw9\" (UniqueName: \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-kube-api-access-4svw9\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:06.891679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.891640 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:06.909881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.909805 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.909881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.909855 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.910247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.909897 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.910247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.909961 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.910247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.910002 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.910247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.910042 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.910247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.910071 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.910247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.910160 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.910247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.910189 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w447\" (UniqueName: \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-kube-api-access-7w447\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.910645 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.910342 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.910645 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.910581 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.911598 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.911547 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.911598 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.911570 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.911822 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.911727 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.912702 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.912680 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.913813 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.913785 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.923626 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.923604 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w447\" (UniqueName: \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-kube-api-access-7w447\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.931473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.931446 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:06.987156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:06.987120 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:07.097331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.097293 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:07.107350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.107317 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:07.156402 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.156323 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd"] Nov 02 17:49:07.164567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.164545 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.179282 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.179260 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd"] Nov 02 17:49:07.197268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.197237 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ea3867-openshift-default-dockercfg-sdtfb\"" Nov 02 17:49:07.212655 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.212625 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.212824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.212676 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcw7\" (UniqueName: \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-kube-api-access-tvcw7\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.212824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.212781 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.212824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.212804 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.212989 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.212967 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.213026 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.213015 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcdd0a3-0820-4007-977e-c0595b979cb7-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.213072 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.213057 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.213145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.213115 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.213224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.213200 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.314685 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.314651 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.314843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.314692 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcdd0a3-0820-4007-977e-c0595b979cb7-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.314843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.314749 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.314843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.314794 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.314843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.314822 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.314843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.314844 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.315118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.315009 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcw7\" (UniqueName: \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-kube-api-access-tvcw7\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.315191 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.315116 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.315191 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.315152 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.315300 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.315190 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.315300 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.315219 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.315300 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.315280 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.315479 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.315457 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcdd0a3-0820-4007-977e-c0595b979cb7-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.315531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.315477 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.317215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.317190 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.317326 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.317294 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.322864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.322840 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k"] Nov 02 17:49:07.324910 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.324882 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:07.326048 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:07.326025 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d182005_2390_4c46_8327_c051ce27933a.slice/crio-603b77dbef38c86a757476d9db0212d06880c9afb2c9652014507c7a56e22141 WatchSource:0}: Error finding container 603b77dbef38c86a757476d9db0212d06880c9afb2c9652014507c7a56e22141: Status 404 returned error can't find the container with id 603b77dbef38c86a757476d9db0212d06880c9afb2c9652014507c7a56e22141 Nov 02 17:49:07.326669 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.326588 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.328292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.328265 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:07.328366 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.328322 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:07.328366 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.328350 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:07.343285 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.343265 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcw7\" (UniqueName: \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-kube-api-access-tvcw7\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.354179 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.354157 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b"] Nov 02 17:49:07.359169 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.359139 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.372406 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.372386 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7b0683-openshift-default-dockercfg-rp64v\"" Nov 02 17:49:07.417905 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.417809 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.418111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.417908 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzq8f\" (UniqueName: \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-kube-api-access-mzq8f\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.418111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.417954 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.418111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.418000 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.418111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.418023 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.418111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.418056 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.418392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.418209 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.418392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.418252 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.418392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.418293 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.446381 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.446346 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng"] Nov 02 17:49:07.479240 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.475013 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:07.479240 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.475123 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:07.479240 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.475166 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:07.484948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.484892 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:07.520462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.520065 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.520462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.520164 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzq8f\" (UniqueName: \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-kube-api-access-mzq8f\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.520462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.520204 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.520462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.520265 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.520462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.520290 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.520462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.520323 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.520462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.520450 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.520866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.520493 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.520866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.520533 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.522382 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.520991 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.522382 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.521250 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.522382 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.521627 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.523997 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.523610 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.529155 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.529037 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b"] Nov 02 17:49:07.529440 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.529378 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.540356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.539697 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.540356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.540186 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.547771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.547708 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzq8f\" (UniqueName: \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-kube-api-access-mzq8f\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.550118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.550020 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.650763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.650673 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:07.687524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.687206 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:07.775516 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.773742 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:07.804957 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.804928 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd"] Nov 02 17:49:07.807532 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:07.807465 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcdd0a3_0820_4007_977e_c0595b979cb7.slice/crio-cba88597dec175a4b4cefb927e74a29386c996dc320952356f4ed69bc7ba9086 WatchSource:0}: Error finding container cba88597dec175a4b4cefb927e74a29386c996dc320952356f4ed69bc7ba9086: Status 404 returned error can't find the container with id cba88597dec175a4b4cefb927e74a29386c996dc320952356f4ed69bc7ba9086 Nov 02 17:49:07.809996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.809961 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:07.810110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.810045 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:07.810181 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.810107 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:07.981683 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.981645 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b"] Nov 02 17:49:07.987102 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:07.987061 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6a1e76_95ae_4ef4_9123_b2518eee56a0.slice/crio-e51ec531812a819fa938bf3bb9701fa69ca698cebd49df504c4f3d7fa1a6f582 WatchSource:0}: Error finding container e51ec531812a819fa938bf3bb9701fa69ca698cebd49df504c4f3d7fa1a6f582: Status 404 returned error can't find the container with id e51ec531812a819fa938bf3bb9701fa69ca698cebd49df504c4f3d7fa1a6f582 Nov 02 17:49:07.989230 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.989203 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:07.989328 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.989267 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:07.989328 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:07.989293 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:08.104481 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.104442 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" event={"ID":"7dcdd0a3-0820-4007-977e-c0595b979cb7","Type":"ContainerStarted","Data":"18a529e39ff27b70b8e6c851b9371f6e245415d73e6fe9d0f7ff3b379fcf9df9"} Nov 02 17:49:08.104658 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.104500 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" event={"ID":"7dcdd0a3-0820-4007-977e-c0595b979cb7","Type":"ContainerStarted","Data":"cba88597dec175a4b4cefb927e74a29386c996dc320952356f4ed69bc7ba9086"} Nov 02 17:49:08.106039 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.105999 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" event={"ID":"2d182005-2390-4c46-8327-c051ce27933a","Type":"ContainerStarted","Data":"eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e"} Nov 02 17:49:08.106198 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.106119 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" event={"ID":"2d182005-2390-4c46-8327-c051ce27933a","Type":"ContainerStarted","Data":"603b77dbef38c86a757476d9db0212d06880c9afb2c9652014507c7a56e22141"} Nov 02 17:49:08.107819 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.107792 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" event={"ID":"4106a2f4-2564-4e87-a040-0c62ecaf2fb8","Type":"ContainerStarted","Data":"5c429c0210d998fc1eb71770f8d81b6e13ef9140951ad66c65965920bbaa3c4b"} Nov 02 17:49:08.108045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.108027 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" event={"ID":"4106a2f4-2564-4e87-a040-0c62ecaf2fb8","Type":"ContainerStarted","Data":"55fb2ffb26612387fb8f6e7586b49b6d1ede5e780655a6250138806b49f839cc"} Nov 02 17:49:08.109759 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.109736 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" event={"ID":"be6a1e76-95ae-4ef4-9123-b2518eee56a0","Type":"ContainerStarted","Data":"0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c"} Nov 02 17:49:08.109922 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.109907 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" event={"ID":"be6a1e76-95ae-4ef4-9123-b2518eee56a0","Type":"ContainerStarted","Data":"e51ec531812a819fa938bf3bb9701fa69ca698cebd49df504c4f3d7fa1a6f582"} Nov 02 17:49:08.127351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.127299 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" podStartSLOduration=1.12728435 podStartE2EDuration="1.12728435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:08.127263403 +0000 UTC m=+545.570346040" watchObservedRunningTime="2026-11-02 17:49:08.12728435 +0000 UTC m=+545.570366986" Nov 02 17:49:08.153658 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.153599 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" podStartSLOduration=2.153581567 podStartE2EDuration="2.153581567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:08.151413951 +0000 UTC m=+545.594496611" watchObservedRunningTime="2026-11-02 17:49:08.153581567 +0000 UTC m=+545.596664205" Nov 02 17:49:08.201139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.201043 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podStartSLOduration=2.201022997 podStartE2EDuration="2.201022997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:08.176267561 +0000 UTC m=+545.619350199" watchObservedRunningTime="2026-11-02 17:49:08.201022997 +0000 UTC m=+545.644105633" Nov 02 17:49:08.201567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.201497 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" podStartSLOduration=1.201488187 podStartE2EDuration="1.201488187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:08.200053341 +0000 UTC m=+545.643135976" watchObservedRunningTime="2026-11-02 17:49:08.201488187 +0000 UTC m=+545.644570899" Nov 02 17:49:08.486331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.486293 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:08.491790 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.491767 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:08.688170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.688142 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:08.987830 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.987791 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:08.995156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:08.995130 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:09.098171 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:09.098138 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:09.103978 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:09.103953 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:09.114818 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:09.114785 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:09.114967 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:09.114847 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:09.114967 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:09.114870 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:09.115518 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:09.115495 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:09.115672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:09.115653 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:09.115860 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:09.115845 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:09.694067 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:09.694036 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:10.063015 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:10.062971 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:10.120500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:10.120470 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:10.121146 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:10.121128 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:10.250690 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:10.250661 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:10.392649 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:10.392563 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:10.598747 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:10.598711 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:10.802582 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:10.802550 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:10.968622 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:10.968586 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:11.275229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:11.275194 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:11.476468 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:11.476424 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:12.471162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:12.471120 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:12.591297 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:12.591266 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:13.502408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:13.502376 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:13.701100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:13.695622 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:14.587561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:14.587520 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:14.702291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:14.702255 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:15.419417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.419372 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:15.534944 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.534910 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:15.541366 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.541319 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn"] Nov 02 17:49:15.546406 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.546380 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:49:15.556196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.556168 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f37d16\"" Nov 02 17:49:15.556368 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.556264 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f37d16-dockercfg-mf8tp\"" Nov 02 17:49:15.562309 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.562267 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn"] Nov 02 17:49:15.562454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.562343 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52"] Nov 02 17:49:15.568377 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.568347 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:49:15.579419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.579389 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52"] Nov 02 17:49:15.652731 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.652680 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b453de3f-2c9c-4705-ab30-b3eae093a0b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn\" (UID: \"b453de3f-2c9c-4705-ab30-b3eae093a0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:49:15.652731 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.652735 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhm2\" (UniqueName: \"kubernetes.io/projected/b453de3f-2c9c-4705-ab30-b3eae093a0b0-kube-api-access-fbhm2\") pod \"payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn\" (UID: \"b453de3f-2c9c-4705-ab30-b3eae093a0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:49:15.653312 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.652770 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rccv\" (UniqueName: \"kubernetes.io/projected/2469cdd9-c68f-477b-9cb8-85aadba1bacd-kube-api-access-9rccv\") pod \"payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52\" (UID: \"2469cdd9-c68f-477b-9cb8-85aadba1bacd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:49:15.653312 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.652849 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2469cdd9-c68f-477b-9cb8-85aadba1bacd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52\" (UID: \"2469cdd9-c68f-477b-9cb8-85aadba1bacd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:49:15.693580 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.693498 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:15.755105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.754717 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b453de3f-2c9c-4705-ab30-b3eae093a0b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn\" (UID: \"b453de3f-2c9c-4705-ab30-b3eae093a0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:49:15.755105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.754771 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhm2\" (UniqueName: \"kubernetes.io/projected/b453de3f-2c9c-4705-ab30-b3eae093a0b0-kube-api-access-fbhm2\") pod \"payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn\" (UID: \"b453de3f-2c9c-4705-ab30-b3eae093a0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:49:15.755105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.754806 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rccv\" (UniqueName: \"kubernetes.io/projected/2469cdd9-c68f-477b-9cb8-85aadba1bacd-kube-api-access-9rccv\") pod \"payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52\" (UID: \"2469cdd9-c68f-477b-9cb8-85aadba1bacd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:49:15.755105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.754884 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2469cdd9-c68f-477b-9cb8-85aadba1bacd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52\" (UID: \"2469cdd9-c68f-477b-9cb8-85aadba1bacd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:49:15.763101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.755746 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2469cdd9-c68f-477b-9cb8-85aadba1bacd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52\" (UID: \"2469cdd9-c68f-477b-9cb8-85aadba1bacd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:49:15.763101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.757001 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b453de3f-2c9c-4705-ab30-b3eae093a0b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn\" (UID: \"b453de3f-2c9c-4705-ab30-b3eae093a0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:49:15.770125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.770040 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhm2\" (UniqueName: \"kubernetes.io/projected/b453de3f-2c9c-4705-ab30-b3eae093a0b0-kube-api-access-fbhm2\") pod \"payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn\" (UID: \"b453de3f-2c9c-4705-ab30-b3eae093a0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:49:15.773705 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.773556 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rccv\" (UniqueName: \"kubernetes.io/projected/2469cdd9-c68f-477b-9cb8-85aadba1bacd-kube-api-access-9rccv\") pod \"payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52\" (UID: \"2469cdd9-c68f-477b-9cb8-85aadba1bacd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:49:15.865610 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.865572 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:49:15.887728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:15.887694 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:49:16.155277 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.155254 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn"] Nov 02 17:49:16.157220 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:16.157188 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb453de3f_2c9c_4705_ab30_b3eae093a0b0.slice/crio-0b7cccc334807c2a96d493f3442e27caa6abd2509f51286ca5df5e8ddd2f1ad1 WatchSource:0}: Error finding container 0b7cccc334807c2a96d493f3442e27caa6abd2509f51286ca5df5e8ddd2f1ad1: Status 404 returned error can't find the container with id 0b7cccc334807c2a96d493f3442e27caa6abd2509f51286ca5df5e8ddd2f1ad1 Nov 02 17:49:16.181747 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.181721 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52"] Nov 02 17:49:16.183937 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:16.183903 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2469cdd9_c68f_477b_9cb8_85aadba1bacd.slice/crio-cc35c366869f6bb171efe41ffbac1a64e17d9d1d950f0ed5ef0017f4e754f596 WatchSource:0}: Error finding container cc35c366869f6bb171efe41ffbac1a64e17d9d1d950f0ed5ef0017f4e754f596: Status 404 returned error can't find the container with id cc35c366869f6bb171efe41ffbac1a64e17d9d1d950f0ed5ef0017f4e754f596 Nov 02 17:49:16.379247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.379208 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:16.402747 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.402715 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b"] Nov 02 17:49:16.407634 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.407603 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:49:16.414489 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.414450 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b"] Nov 02 17:49:16.416949 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.416923 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-9dfa27\"" Nov 02 17:49:16.481597 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.481564 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:16.566147 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.562954 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g95vd\" (UniqueName: \"kubernetes.io/projected/605b73ac-e668-4fc1-ad46-a901f791d92b-kube-api-access-g95vd\") pod \"payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b\" (UID: \"605b73ac-e668-4fc1-ad46-a901f791d92b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:49:16.566147 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.563049 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605b73ac-e668-4fc1-ad46-a901f791d92b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b\" (UID: \"605b73ac-e668-4fc1-ad46-a901f791d92b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:49:16.663686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.663601 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g95vd\" (UniqueName: \"kubernetes.io/projected/605b73ac-e668-4fc1-ad46-a901f791d92b-kube-api-access-g95vd\") pod \"payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b\" (UID: \"605b73ac-e668-4fc1-ad46-a901f791d92b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:49:16.663686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.663669 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605b73ac-e668-4fc1-ad46-a901f791d92b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b\" (UID: \"605b73ac-e668-4fc1-ad46-a901f791d92b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:49:16.664509 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.664490 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605b73ac-e668-4fc1-ad46-a901f791d92b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b\" (UID: \"605b73ac-e668-4fc1-ad46-a901f791d92b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:49:16.677674 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.677647 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g95vd\" (UniqueName: \"kubernetes.io/projected/605b73ac-e668-4fc1-ad46-a901f791d92b-kube-api-access-g95vd\") pod \"payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b\" (UID: \"605b73ac-e668-4fc1-ad46-a901f791d92b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:49:16.724009 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:16.723963 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:49:17.008217 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:17.008188 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b"] Nov 02 17:49:17.012297 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:17.012268 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605b73ac_e668_4fc1_ad46_a901f791d92b.slice/crio-10d0d3ad21eb7fff720713a00a8886407815d0aad2b37e5fe4f6c80dd8fb94bc WatchSource:0}: Error finding container 10d0d3ad21eb7fff720713a00a8886407815d0aad2b37e5fe4f6c80dd8fb94bc: Status 404 returned error can't find the container with id 10d0d3ad21eb7fff720713a00a8886407815d0aad2b37e5fe4f6c80dd8fb94bc Nov 02 17:49:17.157833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:17.157745 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" event={"ID":"b453de3f-2c9c-4705-ab30-b3eae093a0b0","Type":"ContainerStarted","Data":"0b7cccc334807c2a96d493f3442e27caa6abd2509f51286ca5df5e8ddd2f1ad1"} Nov 02 17:49:17.159948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:17.159869 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" event={"ID":"605b73ac-e668-4fc1-ad46-a901f791d92b","Type":"ContainerStarted","Data":"10d0d3ad21eb7fff720713a00a8886407815d0aad2b37e5fe4f6c80dd8fb94bc"} Nov 02 17:49:17.161467 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:17.161442 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" event={"ID":"2469cdd9-c68f-477b-9cb8-85aadba1bacd","Type":"ContainerStarted","Data":"cc35c366869f6bb171efe41ffbac1a64e17d9d1d950f0ed5ef0017f4e754f596"} Nov 02 17:49:17.374373 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:17.374253 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:17.489186 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:17.489097 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:17.621324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:17.621290 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:18.239597 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.239560 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:18.285325 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.285265 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9"] Nov 02 17:49:18.296999 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.296968 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:49:18.307371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.307343 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-7b0683-dockercfg-gjwwz\"" Nov 02 17:49:18.307536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.307428 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-7b0683\"" Nov 02 17:49:18.310878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.310855 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9"] Nov 02 17:49:18.311001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.310914 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh"] Nov 02 17:49:18.317357 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.316171 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:49:18.332299 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.329757 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh"] Nov 02 17:49:18.371242 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.371206 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:18.388936 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.388898 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66cbc84-3c7c-471b-914a-855635642fc8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9\" (UID: \"b66cbc84-3c7c-471b-914a-855635642fc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:49:18.389142 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.389026 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nsc\" (UniqueName: \"kubernetes.io/projected/b66cbc84-3c7c-471b-914a-855635642fc8-kube-api-access-24nsc\") pod \"payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9\" (UID: \"b66cbc84-3c7c-471b-914a-855635642fc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:49:18.514149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.510145 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nsc\" (UniqueName: \"kubernetes.io/projected/b66cbc84-3c7c-471b-914a-855635642fc8-kube-api-access-24nsc\") pod \"payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9\" (UID: \"b66cbc84-3c7c-471b-914a-855635642fc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:49:18.514149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.510271 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c467c116-1975-4e66-ab08-42ec962e170f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh\" (UID: \"c467c116-1975-4e66-ab08-42ec962e170f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:49:18.514149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.510307 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66cbc84-3c7c-471b-914a-855635642fc8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9\" (UID: \"b66cbc84-3c7c-471b-914a-855635642fc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:49:18.514149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.510337 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb727\" (UniqueName: \"kubernetes.io/projected/c467c116-1975-4e66-ab08-42ec962e170f-kube-api-access-rb727\") pod \"payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh\" (UID: \"c467c116-1975-4e66-ab08-42ec962e170f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:49:18.514149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.512766 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66cbc84-3c7c-471b-914a-855635642fc8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9\" (UID: \"b66cbc84-3c7c-471b-914a-855635642fc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:49:18.514529 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.514318 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:18.531689 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.531555 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nsc\" (UniqueName: \"kubernetes.io/projected/b66cbc84-3c7c-471b-914a-855635642fc8-kube-api-access-24nsc\") pod \"payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9\" (UID: \"b66cbc84-3c7c-471b-914a-855635642fc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:49:18.611087 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.611026 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c467c116-1975-4e66-ab08-42ec962e170f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh\" (UID: \"c467c116-1975-4e66-ab08-42ec962e170f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:49:18.611278 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.611114 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb727\" (UniqueName: \"kubernetes.io/projected/c467c116-1975-4e66-ab08-42ec962e170f-kube-api-access-rb727\") pod \"payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh\" (UID: \"c467c116-1975-4e66-ab08-42ec962e170f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:49:18.611934 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.611905 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c467c116-1975-4e66-ab08-42ec962e170f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh\" (UID: \"c467c116-1975-4e66-ab08-42ec962e170f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:49:18.620971 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.620932 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb727\" (UniqueName: \"kubernetes.io/projected/c467c116-1975-4e66-ab08-42ec962e170f-kube-api-access-rb727\") pod \"payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh\" (UID: \"c467c116-1975-4e66-ab08-42ec962e170f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:49:18.623790 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.623752 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:49:18.647409 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.647376 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:49:18.953293 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:18.953233 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc467c116_1975_4e66_ab08_42ec962e170f.slice/crio-251cf644a65cf5b78e18eb05c1af05d9ea7b13166609c85a0d7cb101424605a4 WatchSource:0}: Error finding container 251cf644a65cf5b78e18eb05c1af05d9ea7b13166609c85a0d7cb101424605a4: Status 404 returned error can't find the container with id 251cf644a65cf5b78e18eb05c1af05d9ea7b13166609c85a0d7cb101424605a4 Nov 02 17:49:18.961286 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:18.961159 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66cbc84_3c7c_471b_914a_855635642fc8.slice/crio-d5f729369ee47fc79af7a8799cec49ce38b5028a3cd31c2818ccef1ebbbecf92 WatchSource:0}: Error finding container d5f729369ee47fc79af7a8799cec49ce38b5028a3cd31c2818ccef1ebbbecf92: Status 404 returned error can't find the container with id d5f729369ee47fc79af7a8799cec49ce38b5028a3cd31c2818ccef1ebbbecf92 Nov 02 17:49:18.961286 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.961211 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh"] Nov 02 17:49:18.964538 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:18.964512 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9"] Nov 02 17:49:19.148725 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.148638 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:19.182893 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.182850 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" event={"ID":"b66cbc84-3c7c-471b-914a-855635642fc8","Type":"ContainerStarted","Data":"d5f729369ee47fc79af7a8799cec49ce38b5028a3cd31c2818ccef1ebbbecf92"} Nov 02 17:49:19.184469 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.184402 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" event={"ID":"c467c116-1975-4e66-ab08-42ec962e170f","Type":"ContainerStarted","Data":"251cf644a65cf5b78e18eb05c1af05d9ea7b13166609c85a0d7cb101424605a4"} Nov 02 17:49:19.189191 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.189166 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l"] Nov 02 17:49:19.194784 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.194758 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 17:49:19.203608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.203350 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ea752f\"" Nov 02 17:49:19.208416 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.208360 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l"] Nov 02 17:49:19.256839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.256797 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:19.332534 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.332506 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12043acc-60b0-48d1-9164-127fc5215f28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l\" (UID: \"12043acc-60b0-48d1-9164-127fc5215f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 17:49:19.332636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.332550 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86pt\" (UniqueName: \"kubernetes.io/projected/12043acc-60b0-48d1-9164-127fc5215f28-kube-api-access-t86pt\") pod \"payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l\" (UID: \"12043acc-60b0-48d1-9164-127fc5215f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 17:49:19.353452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.353416 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:19.433375 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.433351 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12043acc-60b0-48d1-9164-127fc5215f28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l\" (UID: \"12043acc-60b0-48d1-9164-127fc5215f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 17:49:19.433484 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.433387 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86pt\" (UniqueName: \"kubernetes.io/projected/12043acc-60b0-48d1-9164-127fc5215f28-kube-api-access-t86pt\") pod \"payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l\" (UID: \"12043acc-60b0-48d1-9164-127fc5215f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 17:49:19.434211 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.434188 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12043acc-60b0-48d1-9164-127fc5215f28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l\" (UID: \"12043acc-60b0-48d1-9164-127fc5215f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 17:49:19.487423 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.486801 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86pt\" (UniqueName: \"kubernetes.io/projected/12043acc-60b0-48d1-9164-127fc5215f28-kube-api-access-t86pt\") pod \"payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l\" (UID: \"12043acc-60b0-48d1-9164-127fc5215f28\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 17:49:19.487423 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.487325 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:19.518544 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.517937 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 17:49:19.827793 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:19.827762 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l"] Nov 02 17:49:19.830452 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:19.830421 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12043acc_60b0_48d1_9164_127fc5215f28.slice/crio-aa717aa1b03da112140be268718989271277ce18fcf978c93b54d1e2140e1907 WatchSource:0}: Error finding container aa717aa1b03da112140be268718989271277ce18fcf978c93b54d1e2140e1907: Status 404 returned error can't find the container with id aa717aa1b03da112140be268718989271277ce18fcf978c93b54d1e2140e1907 Nov 02 17:49:20.192877 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.192826 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" event={"ID":"b453de3f-2c9c-4705-ab30-b3eae093a0b0","Type":"ContainerStarted","Data":"b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a"} Nov 02 17:49:20.193050 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.192964 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:49:20.195233 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.195208 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" event={"ID":"b66cbc84-3c7c-471b-914a-855635642fc8","Type":"ContainerStarted","Data":"3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af"} Nov 02 17:49:20.195377 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.195309 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:49:20.196963 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.196938 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" event={"ID":"12043acc-60b0-48d1-9164-127fc5215f28","Type":"ContainerStarted","Data":"ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57"} Nov 02 17:49:20.197105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.196990 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 17:49:20.197105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.197008 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" event={"ID":"12043acc-60b0-48d1-9164-127fc5215f28","Type":"ContainerStarted","Data":"aa717aa1b03da112140be268718989271277ce18fcf978c93b54d1e2140e1907"} Nov 02 17:49:20.202514 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.202465 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" event={"ID":"605b73ac-e668-4fc1-ad46-a901f791d92b","Type":"ContainerStarted","Data":"cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad"} Nov 02 17:49:20.202838 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.202819 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:49:20.205093 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.205062 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" event={"ID":"c467c116-1975-4e66-ab08-42ec962e170f","Type":"ContainerStarted","Data":"5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003"} Nov 02 17:49:20.205175 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.205138 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:49:20.206889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.206865 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" event={"ID":"2469cdd9-c68f-477b-9cb8-85aadba1bacd","Type":"ContainerStarted","Data":"b57773de42695ac73c95e591f83f2ea173a22f3600d650794563c869c45f92ae"} Nov 02 17:49:20.206985 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.206909 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:49:20.219925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.219884 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" podStartSLOduration=1.466866093 podStartE2EDuration="5.219873421s" totalImagesPullingTime="3.753007328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:20.21702111 +0000 UTC m=+557.660103747" watchObservedRunningTime="2026-11-02 17:49:20.219873421 +0000 UTC m=+557.662956057" Nov 02 17:49:20.246445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.246398 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" podStartSLOduration=1.784252387 podStartE2EDuration="2.246381405s" totalImagesPullingTime="462.129018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:20.244211412 +0000 UTC m=+557.687294040" watchObservedRunningTime="2026-11-02 17:49:20.246381405 +0000 UTC m=+557.689464061" Nov 02 17:49:20.302392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.302358 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:20.313886 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.313838 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" podStartSLOduration=1.900945662 podStartE2EDuration="2.313824024s" totalImagesPullingTime="412.878362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:20.280143335 +0000 UTC m=+557.723225968" watchObservedRunningTime="2026-11-02 17:49:20.313824024 +0000 UTC m=+557.756906660" Nov 02 17:49:20.314197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.314174 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" podStartSLOduration=2.175645095 podStartE2EDuration="5.314165469s" totalImagesPullingTime="3.138520374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:20.312274141 +0000 UTC m=+557.755356793" watchObservedRunningTime="2026-11-02 17:49:20.314165469 +0000 UTC m=+557.757248105" Nov 02 17:49:20.344056 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.343953 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" podStartSLOduration=1.290165057 podStartE2EDuration="4.343941311s" totalImagesPullingTime="3.053776254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:20.342187793 +0000 UTC m=+557.785270430" watchObservedRunningTime="2026-11-02 17:49:20.343941311 +0000 UTC m=+557.787023946" Nov 02 17:49:20.386421 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.386379 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4"] Nov 02 17:49:20.390331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.390308 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:49:20.399441 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.399403 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" podStartSLOduration=1.399390293 podStartE2EDuration="1.399390293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:20.397892177 +0000 UTC m=+557.840974813" watchObservedRunningTime="2026-11-02 17:49:20.399390293 +0000 UTC m=+557.842472929" Nov 02 17:49:20.414182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.414155 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ea3867-dockercfg-s6rrb\"" Nov 02 17:49:20.414413 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.414399 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ea3867\"" Nov 02 17:49:20.454693 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.449355 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvrfd\" (UniqueName: \"kubernetes.io/projected/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-kube-api-access-fvrfd\") pod \"payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4\" (UID: \"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:49:20.454693 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.449908 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4\" (UID: \"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:49:20.457414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.457384 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4"] Nov 02 17:49:20.545205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.545151 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:20.551439 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.551408 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvrfd\" (UniqueName: \"kubernetes.io/projected/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-kube-api-access-fvrfd\") pod \"payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4\" (UID: \"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:49:20.551673 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.551641 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4\" (UID: \"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:49:20.552648 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.552608 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4\" (UID: \"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:49:20.591177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.591137 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvrfd\" (UniqueName: \"kubernetes.io/projected/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-kube-api-access-fvrfd\") pod \"payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4\" (UID: \"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:49:20.704559 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.704483 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:49:20.988601 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:20.988569 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4"] Nov 02 17:49:20.990127 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:20.990095 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3ad3de_693e_4371_8e24_cc5d3d3e7f6a.slice/crio-1decc322f9c7681a1372fd5edbf341d58bf8647cea436afc89ebb34913fd86f5 WatchSource:0}: Error finding container 1decc322f9c7681a1372fd5edbf341d58bf8647cea436afc89ebb34913fd86f5: Status 404 returned error can't find the container with id 1decc322f9c7681a1372fd5edbf341d58bf8647cea436afc89ebb34913fd86f5 Nov 02 17:49:21.214903 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:21.214868 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" event={"ID":"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a","Type":"ContainerStarted","Data":"badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f"} Nov 02 17:49:21.215188 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:21.215158 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" event={"ID":"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a","Type":"ContainerStarted","Data":"1decc322f9c7681a1372fd5edbf341d58bf8647cea436afc89ebb34913fd86f5"} Nov 02 17:49:21.215292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:21.215199 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:49:21.241198 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:21.238617 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" podStartSLOduration=1.238596023 podStartE2EDuration="1.238596023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:21.234858916 +0000 UTC m=+558.677941564" watchObservedRunningTime="2026-11-02 17:49:21.238596023 +0000 UTC m=+558.681678661" Nov 02 17:49:21.311907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:21.311787 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:21.451678 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:21.451580 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:22.327192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.327150 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:22.402547 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.402512 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg"] Nov 02 17:49:22.406791 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.406761 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:49:22.408625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.408600 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg"] Nov 02 17:49:22.411502 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.411481 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-2a295c-dockercfg-d8447\"" Nov 02 17:49:22.411502 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.411482 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-2a295c\"" Nov 02 17:49:22.464664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.464612 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:22.505016 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.504131 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q"] Nov 02 17:49:22.536395 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.536362 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q"] Nov 02 17:49:22.538162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.536621 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:49:22.591001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.590904 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9pj5\" (UniqueName: \"kubernetes.io/projected/378c5f34-0627-4df2-8096-5bb1e582c650-kube-api-access-z9pj5\") pod \"payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg\" (UID: \"378c5f34-0627-4df2-8096-5bb1e582c650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:49:22.591462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.591435 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378c5f34-0627-4df2-8096-5bb1e582c650-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg\" (UID: \"378c5f34-0627-4df2-8096-5bb1e582c650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:49:22.693702 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.693659 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378c5f34-0627-4df2-8096-5bb1e582c650-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg\" (UID: \"378c5f34-0627-4df2-8096-5bb1e582c650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:49:22.698664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.698254 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9pj5\" (UniqueName: \"kubernetes.io/projected/378c5f34-0627-4df2-8096-5bb1e582c650-kube-api-access-z9pj5\") pod \"payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg\" (UID: \"378c5f34-0627-4df2-8096-5bb1e582c650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:49:22.698664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.698409 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhgvw\" (UniqueName: \"kubernetes.io/projected/af852c12-fc47-48e9-ac51-7418a37792d5-kube-api-access-vhgvw\") pod \"payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q\" (UID: \"af852c12-fc47-48e9-ac51-7418a37792d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:49:22.698664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.698455 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af852c12-fc47-48e9-ac51-7418a37792d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q\" (UID: \"af852c12-fc47-48e9-ac51-7418a37792d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:49:22.698664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.698040 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378c5f34-0627-4df2-8096-5bb1e582c650-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg\" (UID: \"378c5f34-0627-4df2-8096-5bb1e582c650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:49:22.707346 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.707306 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9pj5\" (UniqueName: \"kubernetes.io/projected/378c5f34-0627-4df2-8096-5bb1e582c650-kube-api-access-z9pj5\") pod \"payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg\" (UID: \"378c5f34-0627-4df2-8096-5bb1e582c650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:49:22.722219 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.722181 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:49:22.800241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.799747 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhgvw\" (UniqueName: \"kubernetes.io/projected/af852c12-fc47-48e9-ac51-7418a37792d5-kube-api-access-vhgvw\") pod \"payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q\" (UID: \"af852c12-fc47-48e9-ac51-7418a37792d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:49:22.800241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.799785 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af852c12-fc47-48e9-ac51-7418a37792d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q\" (UID: \"af852c12-fc47-48e9-ac51-7418a37792d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:49:22.800573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.800551 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af852c12-fc47-48e9-ac51-7418a37792d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q\" (UID: \"af852c12-fc47-48e9-ac51-7418a37792d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:49:22.808697 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.808668 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhgvw\" (UniqueName: \"kubernetes.io/projected/af852c12-fc47-48e9-ac51-7418a37792d5-kube-api-access-vhgvw\") pod \"payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q\" (UID: \"af852c12-fc47-48e9-ac51-7418a37792d5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:49:22.883756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:22.883390 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:49:23.102129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:23.101339 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg"] Nov 02 17:49:23.234103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:23.230866 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" event={"ID":"378c5f34-0627-4df2-8096-5bb1e582c650","Type":"ContainerStarted","Data":"fe7fb37954d23b96881a49fdff49ad3099464df1dfceb81f402844f45fa78293"} Nov 02 17:49:23.324004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:23.322665 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q"] Nov 02 17:49:23.326000 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:23.325970 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf852c12_fc47_48e9_ac51_7418a37792d5.slice/crio-dc53c1e5ca9201edff4e7acb14fb502dfc5dfdd9fa2f2b7ba84c8059d8afc70b WatchSource:0}: Error finding container dc53c1e5ca9201edff4e7acb14fb502dfc5dfdd9fa2f2b7ba84c8059d8afc70b: Status 404 returned error can't find the container with id dc53c1e5ca9201edff4e7acb14fb502dfc5dfdd9fa2f2b7ba84c8059d8afc70b Nov 02 17:49:23.489194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:23.484763 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:24.238039 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:24.238001 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" event={"ID":"378c5f34-0627-4df2-8096-5bb1e582c650","Type":"ContainerStarted","Data":"5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919"} Nov 02 17:49:24.258511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:24.238162 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:49:24.258511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:24.239810 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" event={"ID":"af852c12-fc47-48e9-ac51-7418a37792d5","Type":"ContainerStarted","Data":"99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926"} Nov 02 17:49:24.258511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:24.239849 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" event={"ID":"af852c12-fc47-48e9-ac51-7418a37792d5","Type":"ContainerStarted","Data":"dc53c1e5ca9201edff4e7acb14fb502dfc5dfdd9fa2f2b7ba84c8059d8afc70b"} Nov 02 17:49:24.258511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:24.239874 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:49:24.283449 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:24.266055 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" podStartSLOduration=2.26603323 podStartE2EDuration="2.26603323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:24.262503233 +0000 UTC m=+561.705585895" watchObservedRunningTime="2026-11-02 17:49:24.26603323 +0000 UTC m=+561.709115868" Nov 02 17:49:24.300631 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:24.300560 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" podStartSLOduration=2.300538442 podStartE2EDuration="2.300538442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:24.295642301 +0000 UTC m=+561.738724938" watchObservedRunningTime="2026-11-02 17:49:24.300538442 +0000 UTC m=+561.743621078" Nov 02 17:49:24.671293 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:24.671199 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:24.923505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:24.923405 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:25.460103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:25.458513 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:25.663100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:25.661523 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:25.768104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:25.767129 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:26.008862 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:26.008784 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:26.121768 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:26.121683 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:26.324432 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:26.324394 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:27.218669 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.218638 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:27.821422 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.821385 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9"] Nov 02 17:49:27.826045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.826025 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:27.838104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.835277 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f37d16\"/\"kube-root-ca.crt\"" Nov 02 17:49:27.838104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.835488 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f37d16\"/\"openshift-service-ca.crt\"" Nov 02 17:49:27.838104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.835788 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f37d16\"/\"simulator-w0-f37d16-kserve-self-signed-certs\"" Nov 02 17:49:27.838104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.835855 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f37d16\"/\"default-dockercfg-dtsf8\"" Nov 02 17:49:27.841050 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.841030 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9"] Nov 02 17:49:27.889476 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.889440 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:27.927912 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.927882 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-dshm\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:27.928116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.927924 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tls-certs\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:27.928116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.928013 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-home\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:27.928116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.928069 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-model-cache\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:27.928244 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.928177 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tmp-dir\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:27.928244 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:27.928213 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x899\" (UniqueName: \"kubernetes.io/projected/3a96b395-eab3-4efb-8b1a-dda4c909f13d-kube-api-access-6x899\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.029922 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.029874 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-dshm\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.030144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.029944 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tls-certs\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.030144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.029981 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-home\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.030144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.030019 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-model-cache\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.030144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.030045 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tmp-dir\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.030635 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.030609 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-home\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.031483 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.030071 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x899\" (UniqueName: \"kubernetes.io/projected/3a96b395-eab3-4efb-8b1a-dda4c909f13d-kube-api-access-6x899\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.031483 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.031360 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tmp-dir\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.032012 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.031982 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-model-cache\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.033413 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.033390 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-dshm\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.035606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.035588 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tls-certs\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.041971 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.041939 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x899\" (UniqueName: \"kubernetes.io/projected/3a96b395-eab3-4efb-8b1a-dda4c909f13d-kube-api-access-6x899\") pod \"simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.174155 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.174050 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:28.190777 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.190742 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:28.593670 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:28.593632 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9"] Nov 02 17:49:28.594113 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:28.593793 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a96b395_eab3_4efb_8b1a_dda4c909f13d.slice/crio-ab8b854edc991188fc4cecdc0a3c1330e3f2457464d8d7ad4734cd956e002395 WatchSource:0}: Error finding container ab8b854edc991188fc4cecdc0a3c1330e3f2457464d8d7ad4734cd956e002395: Status 404 returned error can't find the container with id ab8b854edc991188fc4cecdc0a3c1330e3f2457464d8d7ad4734cd956e002395 Nov 02 17:49:29.208376 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:29.208341 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:29.306724 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:29.306683 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" event={"ID":"3a96b395-eab3-4efb-8b1a-dda4c909f13d","Type":"ContainerStarted","Data":"7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253"} Nov 02 17:49:29.306924 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:29.306731 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" event={"ID":"3a96b395-eab3-4efb-8b1a-dda4c909f13d","Type":"ContainerStarted","Data":"ab8b854edc991188fc4cecdc0a3c1330e3f2457464d8d7ad4734cd956e002395"} Nov 02 17:49:29.334712 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:29.334651 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podStartSLOduration=2.334632331 podStartE2EDuration="2.334632331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:29.334488686 +0000 UTC m=+566.777571324" watchObservedRunningTime="2026-11-02 17:49:29.334632331 +0000 UTC m=+566.777714969" Nov 02 17:49:30.006270 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:30.006229 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:30.192104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:30.191456 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:30.273775 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:30.273640 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:30.574662 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:30.574532 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:30.707254 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:30.699569 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:30.811388 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:30.811291 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:31.165098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:31.163028 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:31.221925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:31.220958 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:49:31.221925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:31.221633 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 17:49:31.222626 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:31.222600 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:49:31.222792 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:31.222674 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:49:31.223703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:31.223379 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:49:31.228525 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:31.228496 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:49:31.268680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:31.268640 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:31.405264 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:31.405222 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:31.655103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:31.642201 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:32.225224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:32.225188 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:49:32.909703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:32.909558 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:33.339304 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.339272 4310 generic.go:356] "Generic (PLEG): container finished" podID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerID="0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c" exitCode=0 Nov 02 17:49:33.339726 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.339393 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" event={"ID":"be6a1e76-95ae-4ef4-9123-b2518eee56a0","Type":"ContainerDied","Data":"0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c"} Nov 02 17:49:33.339774 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.339745 4310 scope.go:122] "RemoveContainer" containerID="0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c" Nov 02 17:49:33.341469 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.341388 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:33.341599 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.341476 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:33.341599 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.341525 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:33.342102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.341972 4310 generic.go:356] "Generic (PLEG): container finished" podID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerID="5c429c0210d998fc1eb71770f8d81b6e13ef9140951ad66c65965920bbaa3c4b" exitCode=0 Nov 02 17:49:33.342102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.342038 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" event={"ID":"4106a2f4-2564-4e87-a040-0c62ecaf2fb8","Type":"ContainerDied","Data":"5c429c0210d998fc1eb71770f8d81b6e13ef9140951ad66c65965920bbaa3c4b"} Nov 02 17:49:33.367322 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.356263 4310 scope.go:122] "RemoveContainer" containerID="5c429c0210d998fc1eb71770f8d81b6e13ef9140951ad66c65965920bbaa3c4b" Nov 02 17:49:33.376926 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.376885 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:33.377132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.376961 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:33.377132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.377006 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:33.534047 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:49:33.533980 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:49:33.847160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:33.842350 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:34.015839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:34.015804 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:34.424103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:34.422596 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" event={"ID":"4106a2f4-2564-4e87-a040-0c62ecaf2fb8","Type":"ContainerStarted","Data":"b0f5b83f1498ae55e97eb2d8e2011edd938ff1c1177e55378836b52fc7556f06"} Nov 02 17:49:34.432942 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:34.426710 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" event={"ID":"be6a1e76-95ae-4ef4-9123-b2518eee56a0","Type":"ContainerStarted","Data":"de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957"} Nov 02 17:49:34.692726 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:34.692628 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:34.704112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:34.702227 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 17:49:34.704112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:34.702307 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 17:49:35.104108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:35.103725 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:35.104108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:35.103911 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 17:49:35.104108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:35.103957 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 17:49:35.138427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:35.138396 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:35.274106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:35.269460 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:49:35.274300 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:35.274225 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:49:35.695141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:35.694440 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 17:49:35.695141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:35.694499 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 17:49:36.100179 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:36.100141 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 17:49:36.100467 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:36.100433 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 17:49:36.249750 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:36.249638 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:36.399543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:36.399340 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:36.705169 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:36.704290 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 17:49:36.705169 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:36.704353 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 17:49:37.101088 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:37.100425 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 17:49:37.101088 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:37.100487 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 17:49:37.204767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:37.204722 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:37.299007 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:37.298957 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:37.689095 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:37.688436 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 17:49:37.689095 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:37.688492 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 17:49:38.098542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:38.098495 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 17:49:38.099094 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:38.098569 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 17:49:38.194609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:38.194241 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:38.207978 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:38.207942 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:38.273106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:38.272611 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:38.506354 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:38.506323 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:49:38.702988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:38.698294 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 17:49:38.702988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:38.698355 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 17:49:39.098568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:39.098522 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 17:49:39.099227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:39.099186 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 17:49:39.115215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:39.115175 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:39.205916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:39.203581 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:39.697101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:39.696340 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 17:49:39.697101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:39.696399 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 17:49:39.877367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:39.876416 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:52376->10.133.0.41:15021: read: connection reset by peer" start-of-body= Nov 02 17:49:39.877367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:39.876498 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:52376->10.133.0.41:15021: read: connection reset by peer" Nov 02 17:49:39.886580 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:39.886549 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:40.120318 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.120233 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:40.158236 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.158196 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:49:40.158405 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.158244 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:49:40.223834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.223798 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:40.507174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.507141 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:40.508742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.508719 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:49:40.511380 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.511346 4310 generic.go:356] "Generic (PLEG): container finished" podID="2d182005-2390-4c46-8327-c051ce27933a" containerID="eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e" exitCode=0 Nov 02 17:49:40.511597 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.511428 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" event={"ID":"2d182005-2390-4c46-8327-c051ce27933a","Type":"ContainerDied","Data":"eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e"} Nov 02 17:49:40.511953 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.511933 4310 scope.go:122] "RemoveContainer" containerID="eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e" Nov 02 17:49:40.515194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.514341 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:40.515194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.514453 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:40.515194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.514504 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:40.696700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.695863 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:40.697653 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:40.697629 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:49:41.038566 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.038523 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:41.138111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.138056 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:41.518917 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.518883 4310 generic.go:356] "Generic (PLEG): container finished" podID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerID="18a529e39ff27b70b8e6c851b9371f6e245415d73e6fe9d0f7ff3b379fcf9df9" exitCode=0 Nov 02 17:49:41.519167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.518964 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" event={"ID":"7dcdd0a3-0820-4007-977e-c0595b979cb7","Type":"ContainerDied","Data":"18a529e39ff27b70b8e6c851b9371f6e245415d73e6fe9d0f7ff3b379fcf9df9"} Nov 02 17:49:41.519466 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.519442 4310 scope.go:122] "RemoveContainer" containerID="18a529e39ff27b70b8e6c851b9371f6e245415d73e6fe9d0f7ff3b379fcf9df9" Nov 02 17:49:41.520983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.520946 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:41.521072 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.521026 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:41.521072 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.521063 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:49:41.521210 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.521104 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" event={"ID":"2d182005-2390-4c46-8327-c051ce27933a","Type":"ContainerStarted","Data":"fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773"} Nov 02 17:49:41.988073 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.988042 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:41.988289 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.988247 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:49:41.988289 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.988281 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 17:49:41.993240 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:41.993201 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:42.499099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:42.498525 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:42.529654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:42.529617 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" event={"ID":"7dcdd0a3-0820-4007-977e-c0595b979cb7","Type":"ContainerStarted","Data":"740b0c0155ff13bdb5d7b597ca33fdb3baaa1933df177fdda2a156fcfc7e30dd"} Nov 02 17:49:42.874160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:42.863444 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:42.994102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:42.993513 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:49:42.994102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:42.993579 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 17:49:43.491647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:43.491603 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:43.492054 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:43.492033 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 17:49:43.492291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:43.492243 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 17:49:43.823515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:43.823420 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:43.989102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:43.988478 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:49:43.989102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:43.988539 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 17:49:44.488556 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:44.488515 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 17:49:44.489409 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:44.489377 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 17:49:44.767152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:44.766607 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:44.990570 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:44.990528 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:49:44.991618 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:44.991187 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 17:49:45.489174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:45.488601 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 17:49:45.489174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:45.488665 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 17:49:45.697103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:45.696716 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:45.906934 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:45.906852 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:45.989545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:45.989508 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:49:45.989845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:45.989814 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 17:49:46.492644 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:46.492604 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 17:49:46.493299 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:46.493264 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 17:49:46.808748 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:46.808667 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:46.941410 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:46.941357 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:47.077104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:47.070001 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:47.421399 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:47.421310 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:47.421796 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:47.421767 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:47.422724 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:47.422700 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 17:49:47.485633 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:47.485601 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 17:49:47.485854 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:47.485674 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 17:49:47.861728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:47.861690 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:48.490192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:48.490164 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:48.490490 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:48.490462 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:48.491168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:48.491153 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:49:48.509331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:48.509270 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:49.159778 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:49.159738 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:49.912047 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:49.912010 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:50.577180 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:50.577143 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:51.272763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:51.272726 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:51.977729 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:51.977693 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:52.175354 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:52.175314 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:52.955217 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:52.955182 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:53.681960 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:53.681930 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:54.447815 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:54.447777 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:55.086413 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.086355 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:55.732053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.732015 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:55.884814 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.884778 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf"] Nov 02 17:49:55.888920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.888896 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:55.898807 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.898733 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f37d16\"/\"premium-w0-f37d16-kserve-self-signed-certs\"" Nov 02 17:49:55.903477 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.903457 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf"] Nov 02 17:49:55.947193 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.947161 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:55.972395 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.972366 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-tmp-dir\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:55.972522 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.972415 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-dshm\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:55.972522 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.972484 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a87b59-04df-4013-8602-e50f5ce2842a-tls-certs\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:55.972656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.972638 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-model-cache\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:55.972701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.972671 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-home\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:55.972701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:55.972694 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnwc\" (UniqueName: \"kubernetes.io/projected/89a87b59-04df-4013-8602-e50f5ce2842a-kube-api-access-jtnwc\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.074379 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.074283 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-model-cache\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.074379 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.074328 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-home\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.074379 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.074351 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnwc\" (UniqueName: \"kubernetes.io/projected/89a87b59-04df-4013-8602-e50f5ce2842a-kube-api-access-jtnwc\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.074379 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.074372 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-tmp-dir\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.074683 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.074407 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-dshm\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.074683 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.074526 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a87b59-04df-4013-8602-e50f5ce2842a-tls-certs\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.074794 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.074724 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-model-cache\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.074794 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.074745 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-tmp-dir\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.074900 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.074788 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-home\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.076727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.076701 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-dshm\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.077090 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.077058 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a87b59-04df-4013-8602-e50f5ce2842a-tls-certs\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.083102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.083066 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnwc\" (UniqueName: \"kubernetes.io/projected/89a87b59-04df-4013-8602-e50f5ce2842a-kube-api-access-jtnwc\") pod \"premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.204249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.204206 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:56.487396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.487366 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf"] Nov 02 17:49:56.489631 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:49:56.489604 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89a87b59_04df_4013_8602_e50f5ce2842a.slice/crio-db79a730694070d7f6108c2eaca497a4192b89bbb0b87cf80c1adee2d727f2c8 WatchSource:0}: Error finding container db79a730694070d7f6108c2eaca497a4192b89bbb0b87cf80c1adee2d727f2c8: Status 404 returned error can't find the container with id db79a730694070d7f6108c2eaca497a4192b89bbb0b87cf80c1adee2d727f2c8 Nov 02 17:49:56.555823 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.555792 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:56.642138 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.642073 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" event={"ID":"89a87b59-04df-4013-8602-e50f5ce2842a","Type":"ContainerStarted","Data":"93325fa53b7512224a896f193fe2fe5191d5e1bfd2b696b8637d63a9316da83a"} Nov 02 17:49:56.642295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.642151 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" event={"ID":"89a87b59-04df-4013-8602-e50f5ce2842a","Type":"ContainerStarted","Data":"db79a730694070d7f6108c2eaca497a4192b89bbb0b87cf80c1adee2d727f2c8"} Nov 02 17:49:56.661517 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:56.661464 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podStartSLOduration=1.661448461 podStartE2EDuration="1.661448461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:56.658914171 +0000 UTC m=+594.101996808" watchObservedRunningTime="2026-11-02 17:49:56.661448461 +0000 UTC m=+594.104531096" Nov 02 17:49:57.204994 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:57.204954 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:49:57.569874 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:57.569839 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:57.669125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:57.669066 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:58.240705 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:58.240672 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:58.854691 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:58.854660 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:49:59.461328 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:49:59.461291 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:00.078401 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:00.078366 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:00.763639 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:00.763593 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:01.028780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:01.028699 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:01.451278 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:01.451200 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:02.333295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:02.333248 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:02.902765 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:02.902733 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:03.007562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:03.007521 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:03.524055 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:50:03.523998 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:50:03.775005 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:03.774919 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:04.463727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:04.463691 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:05.062705 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:05.062670 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:05.750714 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:05.750684 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:06.204457 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.204381 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:50:06.220220 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.220194 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:50:06.362979 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.362940 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:06.673481 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.673449 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp"] Nov 02 17:50:06.678477 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.678455 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.686338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.686310 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f37d16-openshift-default-dockercfg-5tcm2\"" Nov 02 17:50:06.688008 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.687986 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp"] Nov 02 17:50:06.716873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.716848 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:50:06.802952 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.802917 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.803166 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.803025 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.803166 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.803107 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90556d84-e481-4860-ace0-12f84117cc76-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.803299 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.803277 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.805140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.803920 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90556d84-e481-4860-ace0-12f84117cc76-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.805140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.804252 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.805140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.804335 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.805140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.804379 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4vf\" (UniqueName: \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-kube-api-access-hz4vf\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.805140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.804444 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.905302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.905263 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.905486 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.905327 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4vf\" (UniqueName: \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-kube-api-access-hz4vf\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.905486 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.905367 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.905486 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.905438 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.905636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.905511 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.905636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.905538 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90556d84-e481-4860-ace0-12f84117cc76-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.905636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.905580 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.905636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.905622 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90556d84-e481-4860-ace0-12f84117cc76-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.905865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.905654 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.905865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.905772 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.906205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.906178 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.906342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.906236 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.906342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.906250 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.906563 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.906541 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90556d84-e481-4860-ace0-12f84117cc76-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.907808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.907789 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.908042 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.908023 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90556d84-e481-4860-ace0-12f84117cc76-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.913416 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.913395 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.913553 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.913529 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4vf\" (UniqueName: \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-kube-api-access-hz4vf\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:06.999030 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:06.998995 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:07.010357 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.010330 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5"] Nov 02 17:50:07.014510 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.014387 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.018057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.018037 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bb4ecb-openshift-default-dockercfg-j9575\"" Nov 02 17:50:07.027402 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.027376 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5"] Nov 02 17:50:07.052139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.052109 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:07.137121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.131021 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.137121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.131090 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zfj2\" (UniqueName: \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-kube-api-access-6zfj2\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.137121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.131117 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.137121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.131182 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.137121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.131213 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.137121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.131269 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.137121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.131307 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.137121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.131343 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.137121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.131372 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.232800 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.232762 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233255 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.232810 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233255 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.232931 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233255 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.233039 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233255 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.233066 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zfj2\" (UniqueName: \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-kube-api-access-6zfj2\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233255 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.233126 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233255 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.233209 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233255 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.233250 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.233270 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.233329 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.233338 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233729 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.233649 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233729 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.233652 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.233729 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.233719 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.235443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.235414 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.235662 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.235642 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.241152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.241126 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.241504 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.241484 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zfj2\" (UniqueName: \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-kube-api-access-6zfj2\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.298976 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.298951 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp"] Nov 02 17:50:07.300413 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:50:07.300368 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90556d84_e481_4860_ace0_12f84117cc76.slice/crio-0f2fa2cf2f808835a9c1248042b7f85bda75b9b4974005d5995c522ff283aa71 WatchSource:0}: Error finding container 0f2fa2cf2f808835a9c1248042b7f85bda75b9b4974005d5995c522ff283aa71: Status 404 returned error can't find the container with id 0f2fa2cf2f808835a9c1248042b7f85bda75b9b4974005d5995c522ff283aa71 Nov 02 17:50:07.303044 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.303008 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:07.303127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.303115 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:07.303172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.303146 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:07.357793 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.357768 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:07.644270 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.644236 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5"] Nov 02 17:50:07.646067 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:50:07.646032 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4447e85f_4043_4e1b_b4ab_e9f36fd1438c.slice/crio-0b5a3fff6941adcfe12bccf5c548c57091fc59544b300eaad9a3566fb7cad0cf WatchSource:0}: Error finding container 0b5a3fff6941adcfe12bccf5c548c57091fc59544b300eaad9a3566fb7cad0cf: Status 404 returned error can't find the container with id 0b5a3fff6941adcfe12bccf5c548c57091fc59544b300eaad9a3566fb7cad0cf Nov 02 17:50:07.648791 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.648749 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:07.648886 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.648844 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:07.648939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.648893 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:07.707756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.707721 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" event={"ID":"4447e85f-4043-4e1b-b4ab-e9f36fd1438c","Type":"ContainerStarted","Data":"0b5a3fff6941adcfe12bccf5c548c57091fc59544b300eaad9a3566fb7cad0cf"} Nov 02 17:50:07.709529 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.709502 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" event={"ID":"90556d84-e481-4860-ace0-12f84117cc76","Type":"ContainerStarted","Data":"907cfa650008c6c144073860e8ff900e24cdb03401a372fb84b6d84d7d92b5fa"} Nov 02 17:50:07.709664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.709552 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" event={"ID":"90556d84-e481-4860-ace0-12f84117cc76","Type":"ContainerStarted","Data":"0f2fa2cf2f808835a9c1248042b7f85bda75b9b4974005d5995c522ff283aa71"} Nov 02 17:50:07.734291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.734233 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" podStartSLOduration=1.73421534 podStartE2EDuration="1.73421534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:07.733633792 +0000 UTC m=+605.176716427" watchObservedRunningTime="2026-11-02 17:50:07.73421534 +0000 UTC m=+605.177297977" Nov 02 17:50:07.959127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.958958 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:07.999418 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:07.999383 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:08.726365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:08.726331 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" event={"ID":"4447e85f-4043-4e1b-b4ab-e9f36fd1438c","Type":"ContainerStarted","Data":"cfc57f5019d0505b8ec132bf473a754b51a4a9420410d89f4054c93784c47b4a"} Nov 02 17:50:08.983010 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:08.982889 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:09.000669 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:09.000628 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:50:09.000845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:09.000683 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:50:09.366280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:09.366186 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:09.366602 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:09.366579 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 17:50:09.366756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:09.366733 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 17:50:09.792103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:09.777544 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:09.999682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:09.999602 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:50:09.999682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:09.999668 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:50:10.369101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:10.363513 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 17:50:10.369101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:10.363574 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 17:50:10.585102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:10.583620 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:10.999693 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:10.999652 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:50:11.000112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:10.999715 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:50:11.365279 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:11.365188 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:11.365602 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:11.365576 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 17:50:11.365786 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:11.365763 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 17:50:11.467098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:11.466855 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:11.999635 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:11.999593 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:50:11.999789 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:11.999661 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:50:12.365173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:12.360472 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 17:50:12.365173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:12.360535 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 17:50:12.435393 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:12.435351 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:12.539071 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:12.539025 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:12.999663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:12.999623 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:50:12.999942 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:12.999915 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:50:13.362159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:13.361696 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 17:50:13.362159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:13.361758 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 17:50:13.481699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:13.481659 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:14.000070 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:14.000032 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 17:50:14.000489 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:14.000101 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 17:50:14.358508 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:14.358409 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 17:50:14.358508 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:14.358479 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 17:50:14.382725 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:14.382690 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:15.004759 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:15.004728 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:15.005278 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:15.005194 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:15.005906 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:15.005889 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:50:15.027284 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:15.027228 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" podStartSLOduration=9.027213981 podStartE2EDuration="9.027213981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:08.762753386 +0000 UTC m=+606.205836012" watchObservedRunningTime="2026-11-02 17:50:15.027213981 +0000 UTC m=+612.470296617" Nov 02 17:50:15.124146 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:15.124117 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:15.364012 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:15.363924 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:15.364360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:15.364337 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:15.365024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:15.365004 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 17:50:15.766781 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:15.766742 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:16.437371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:16.437334 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:16.670408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:16.670373 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:17.360338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:17.360303 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:18.156672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:18.156640 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:18.848273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:18.848237 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:19.196404 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:19.196325 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:19.829598 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:19.829563 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:20.562214 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:20.562177 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:21.283894 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:21.283855 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:22.142068 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.142031 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc"] Nov 02 17:50:22.170475 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.170438 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.171588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.171467 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc"] Nov 02 17:50:22.178229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.178205 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2a295c-openshift-default-dockercfg-kqbfr\"" Nov 02 17:50:22.207459 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.207428 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2v5\" (UniqueName: \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-kube-api-access-kp2v5\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.207643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.207476 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.207643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.207552 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.207643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.207622 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.207817 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.207653 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.207817 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.207698 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed79116-9b99-4427-814f-e8af212f5126-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.207817 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.207735 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.207817 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.207765 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed79116-9b99-4427-814f-e8af212f5126-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.207817 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.207809 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.309336 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.309294 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.309542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.309391 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2v5\" (UniqueName: \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-kube-api-access-kp2v5\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.309542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.309430 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.309542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.309521 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.309719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.309606 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.309719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.309646 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.309719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.309693 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed79116-9b99-4427-814f-e8af212f5126-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.309872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.309728 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.309872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.309759 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed79116-9b99-4427-814f-e8af212f5126-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.309999 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.309957 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.310055 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.310027 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.310283 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.310258 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.310361 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.310300 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.310606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.310584 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed79116-9b99-4427-814f-e8af212f5126-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.312662 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.312644 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.313436 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.313413 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed79116-9b99-4427-814f-e8af212f5126-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.317352 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.317324 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2v5\" (UniqueName: \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-kube-api-access-kp2v5\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.318347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.318327 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.321700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.321680 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:22.501104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.500889 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:22.863005 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.862971 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc"] Nov 02 17:50:22.864769 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:50:22.864727 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed79116_9b99_4427_814f_e8af212f5126.slice/crio-7321115f96a08a8a8dc156652360e311126efdf86020f5c2c01223f3e27c5484 WatchSource:0}: Error finding container 7321115f96a08a8a8dc156652360e311126efdf86020f5c2c01223f3e27c5484: Status 404 returned error can't find the container with id 7321115f96a08a8a8dc156652360e311126efdf86020f5c2c01223f3e27c5484 Nov 02 17:50:22.867589 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.867549 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:22.867699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.867639 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:22.867699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.867690 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:22.895395 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:22.895366 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" event={"ID":"aed79116-9b99-4427-814f-e8af212f5126","Type":"ContainerStarted","Data":"7321115f96a08a8a8dc156652360e311126efdf86020f5c2c01223f3e27c5484"} Nov 02 17:50:23.163436 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:23.163340 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:23.870132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:23.870097 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:23.902269 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:23.902231 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" event={"ID":"aed79116-9b99-4427-814f-e8af212f5126","Type":"ContainerStarted","Data":"8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4"} Nov 02 17:50:23.935270 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:23.929017 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" podStartSLOduration=1.9289961020000002 podStartE2EDuration="1.928996102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:23.927977918 +0000 UTC m=+621.371060572" watchObservedRunningTime="2026-11-02 17:50:23.928996102 +0000 UTC m=+621.372078751" Nov 02 17:50:24.502069 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:24.502029 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:24.504142 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:24.504116 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:50:24.504290 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:24.504177 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" podUID="aed79116-9b99-4427-814f-e8af212f5126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 17:50:24.567782 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:24.567744 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:25.271127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:25.264440 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:25.502365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:25.502328 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:50:25.502872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:25.502381 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" podUID="aed79116-9b99-4427-814f-e8af212f5126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 17:50:25.967102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:25.963556 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:26.502177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:26.502143 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:50:26.502356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:26.502197 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" podUID="aed79116-9b99-4427-814f-e8af212f5126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 17:50:26.660103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:26.653005 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:27.394273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:27.394233 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:27.502697 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:27.502661 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:50:27.502896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:27.502769 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" podUID="aed79116-9b99-4427-814f-e8af212f5126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 17:50:28.115496 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:28.115462 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:28.502263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:28.502228 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 17:50:28.502469 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:28.502293 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" podUID="aed79116-9b99-4427-814f-e8af212f5126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 17:50:28.798106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:28.797947 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:29.479397 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:29.479356 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:29.682308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:29.682279 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:29.683675 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:29.683649 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:29.684918 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:29.684895 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:50:30.460548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:30.460511 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:31.152248 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:31.152206 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:31.758583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:31.758553 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:32.469808 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:32.469773 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:33.165720 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:33.165676 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:33.524030 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:50:33.523996 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:50:33.860473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:33.860389 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:34.634649 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:34.634617 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:35.723809 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:35.723758 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:36.997401 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:36.997369 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:37.875730 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:37.875698 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:38.682570 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:38.682534 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:39.388045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:39.388010 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:40.555287 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:40.555252 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:41.462231 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:41.462188 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:42.247278 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:42.247248 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:43.070215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:43.070184 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:43.854928 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:43.854888 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:44.562065 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:44.562031 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:45.246169 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:45.246134 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:45.736395 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:45.736352 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:45.950517 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:45.950483 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:46.562275 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:46.562242 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:47.260630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:47.260595 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:47.943552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:47.943498 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:48.554812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:48.554752 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:49.157332 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:49.157294 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:49.830714 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:49.830661 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:50.488467 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:50.488427 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:51.165705 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.165671 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:51.735315 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.735274 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq"] Nov 02 17:50:51.739443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.739423 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.759411 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.759351 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq"] Nov 02 17:50:51.793022 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.792986 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.793272 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.793243 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.793411 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.793382 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.793502 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.793483 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.793567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.793552 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zgb\" (UniqueName: \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-kube-api-access-x8zgb\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.793632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.793596 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62cfdc9-79e3-4568-a547-6caddc29126c-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.793686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.793670 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.793755 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.793730 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.793868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.793806 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.866088 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.866038 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:51.894685 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.894651 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zgb\" (UniqueName: \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-kube-api-access-x8zgb\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.894873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.894699 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62cfdc9-79e3-4568-a547-6caddc29126c-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.894873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.894723 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.894873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.894767 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.894873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.894784 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.894873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.894812 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.895196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.894908 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.895196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.894965 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.895196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.895009 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.895356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.895193 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.895356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.895304 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.895471 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.895437 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.895471 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.895462 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.895587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.895563 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62cfdc9-79e3-4568-a547-6caddc29126c-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.897506 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.897484 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.897619 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.897538 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.902873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.902850 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:51.903046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:51.903024 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zgb\" (UniqueName: \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-kube-api-access-x8zgb\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:52.067960 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.067867 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:52.086018 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.082863 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn"] Nov 02 17:50:52.091196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.091047 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.097867 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.097823 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn"] Nov 02 17:50:52.199187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.199142 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.199350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.199291 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rtz\" (UniqueName: \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-kube-api-access-l4rtz\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.199406 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.199376 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13401d6a-300b-4bee-beef-fd0fb068e112-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.199512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.199495 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.199573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.199526 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.199640 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.199622 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.199679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.199647 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.199679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.199667 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.199771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.199685 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13401d6a-300b-4bee-beef-fd0fb068e112-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301093 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301053 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rtz\" (UniqueName: \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-kube-api-access-l4rtz\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301131 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13401d6a-300b-4bee-beef-fd0fb068e112-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301185 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301201 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301268 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301292 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301321 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301397 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13401d6a-300b-4bee-beef-fd0fb068e112-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301601 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301466 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301700 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301829 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.301937 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.301866 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.302236 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.302208 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.302396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.302378 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13401d6a-300b-4bee-beef-fd0fb068e112-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.303828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.303807 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.303912 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.303807 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13401d6a-300b-4bee-beef-fd0fb068e112-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.310554 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.310515 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.311214 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.311190 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rtz\" (UniqueName: \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-kube-api-access-l4rtz\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.364179 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.364150 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq"] Nov 02 17:50:52.365448 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:50:52.365409 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62cfdc9_79e3_4568_a547_6caddc29126c.slice/crio-ce5a4b6851baa6b0a75e28d4da42bd4916029844e0ad50cd2d49c2097ba90c7f WatchSource:0}: Error finding container ce5a4b6851baa6b0a75e28d4da42bd4916029844e0ad50cd2d49c2097ba90c7f: Status 404 returned error can't find the container with id ce5a4b6851baa6b0a75e28d4da42bd4916029844e0ad50cd2d49c2097ba90c7f Nov 02 17:50:52.367944 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.367896 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:52.368023 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.367983 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:52.368023 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.368013 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:52.417972 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.417947 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:52.663820 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.663730 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:52.703889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.703865 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn"] Nov 02 17:50:52.706231 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:50:52.706201 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13401d6a_300b_4bee_beef_fd0fb068e112.slice/crio-66761905a22d48de32eff26d72a866c0aa50067f2b692073679c78a2cce08aee WatchSource:0}: Error finding container 66761905a22d48de32eff26d72a866c0aa50067f2b692073679c78a2cce08aee: Status 404 returned error can't find the container with id 66761905a22d48de32eff26d72a866c0aa50067f2b692073679c78a2cce08aee Nov 02 17:50:52.709221 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.709179 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:52.709308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.709275 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:52.709344 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:52.709316 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:50:53.098818 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:53.098784 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" event={"ID":"e62cfdc9-79e3-4568-a547-6caddc29126c","Type":"ContainerStarted","Data":"7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2"} Nov 02 17:50:53.099400 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:53.098846 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" event={"ID":"e62cfdc9-79e3-4568-a547-6caddc29126c","Type":"ContainerStarted","Data":"ce5a4b6851baa6b0a75e28d4da42bd4916029844e0ad50cd2d49c2097ba90c7f"} Nov 02 17:50:53.100452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:53.100426 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" event={"ID":"13401d6a-300b-4bee-beef-fd0fb068e112","Type":"ContainerStarted","Data":"484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731"} Nov 02 17:50:53.100602 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:53.100467 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" event={"ID":"13401d6a-300b-4bee-beef-fd0fb068e112","Type":"ContainerStarted","Data":"66761905a22d48de32eff26d72a866c0aa50067f2b692073679c78a2cce08aee"} Nov 02 17:50:53.120628 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:53.120576 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" podStartSLOduration=2.120562084 podStartE2EDuration="2.120562084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:53.118290249 +0000 UTC m=+650.561372901" watchObservedRunningTime="2026-11-02 17:50:53.120562084 +0000 UTC m=+650.563644719" Nov 02 17:50:53.139906 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:53.139729 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podStartSLOduration=1.1397092500000001 podStartE2EDuration="1.13970925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:53.139206618 +0000 UTC m=+650.582289255" watchObservedRunningTime="2026-11-02 17:50:53.13970925 +0000 UTC m=+650.582791889" Nov 02 17:50:53.418648 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:53.418569 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:50:53.419949 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:53.419923 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:50:53.420142 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:53.419969 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:50:53.599726 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:53.599692 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:54.088106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:54.084448 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:50:54.094102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:54.093260 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 17:50:54.094102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:54.093339 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 17:50:54.418742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:54.418654 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:50:54.418742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:54.418716 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:50:54.593756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:54.593666 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:55.111244 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:55.110483 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 17:50:55.111244 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:55.110548 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 17:50:55.418864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:55.418762 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:50:55.418864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:55.418822 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:50:55.682156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:55.681914 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:56.084746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:56.084709 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 17:50:56.085013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:56.084980 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 17:50:56.418578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:56.418497 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:50:56.418578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:56.418548 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:50:56.493008 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:56.492973 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:57.077484 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:57.077444 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 17:50:57.077780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:57.077750 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 17:50:57.393496 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:57.393401 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:57.418975 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:57.418940 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:50:57.419182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:57.419000 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:50:58.073539 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:58.073500 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 17:50:58.074183 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:58.074148 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 17:50:58.199109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:58.197896 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:58.418987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:58.418898 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:50:58.418987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:58.418951 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:50:58.981932 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:58.981893 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:50:59.069602 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:59.069563 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 17:50:59.069899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:59.069868 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 17:50:59.418754 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:59.418658 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 17:50:59.418754 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:59.418714 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 17:50:59.790462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:50:59.790428 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:00.073718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:00.073633 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:51:00.074017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:00.073996 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:51:00.074576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:00.074558 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:51:00.423981 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:00.423892 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:51:00.424381 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:00.424343 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:51:00.425038 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:00.425018 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 17:51:00.453512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:00.453469 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:01.169591 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:01.169555 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:01.868139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:01.868071 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:02.598421 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:02.598375 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:03.263434 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:03.263396 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:03.524212 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:51:03.524131 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:51:04.136398 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:04.136365 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:04.893434 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:04.893403 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:05.701745 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:05.701711 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:05.840314 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:05.840284 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx"] Nov 02 17:51:05.845456 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:05.845432 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:05.850552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:05.850526 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:51:05.850700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:05.850582 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:51:05.852875 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:05.852844 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx"] Nov 02 17:51:05.929370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:05.929331 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea0014a1-r7pxx\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:05.929370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:05.929372 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb6dj\" (UniqueName: \"kubernetes.io/projected/43996d70-c3d6-453f-bc7a-67248670d592-kube-api-access-cb6dj\") pod \"maas-api-revoke-sub-ea0014a1-r7pxx\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:05.929781 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:05.929578 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea0014a1-r7pxx\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:06.031170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.031122 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea0014a1-r7pxx\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:06.031376 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.031259 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea0014a1-r7pxx\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:06.031376 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.031286 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb6dj\" (UniqueName: \"kubernetes.io/projected/43996d70-c3d6-453f-bc7a-67248670d592-kube-api-access-cb6dj\") pod \"maas-api-revoke-sub-ea0014a1-r7pxx\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:06.031889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.031867 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea0014a1-r7pxx\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:06.033565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.033541 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea0014a1-r7pxx\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:06.040356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.040331 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb6dj\" (UniqueName: \"kubernetes.io/projected/43996d70-c3d6-453f-bc7a-67248670d592-kube-api-access-cb6dj\") pod \"maas-api-revoke-sub-ea0014a1-r7pxx\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:06.162591 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.162556 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:06.275661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.275628 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k"] Nov 02 17:51:06.280746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.280720 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:06.287291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.287226 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k"] Nov 02 17:51:06.335810 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.335781 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f2db06-rgc9k\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:06.335973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.335832 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f2db06-rgc9k\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:06.335973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.335852 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwgd\" (UniqueName: \"kubernetes.io/projected/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-kube-api-access-zvwgd\") pod \"maas-api-revoke-sub-04f2db06-rgc9k\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:06.437103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.437045 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f2db06-rgc9k\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:06.437272 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.437128 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f2db06-rgc9k\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:06.437272 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.437180 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwgd\" (UniqueName: \"kubernetes.io/projected/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-kube-api-access-zvwgd\") pod \"maas-api-revoke-sub-04f2db06-rgc9k\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:06.437990 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.437928 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f2db06-rgc9k\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:06.439637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.439615 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f2db06-rgc9k\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:06.439637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.439626 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx"] Nov 02 17:51:06.440923 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:06.440891 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43996d70_c3d6_453f_bc7a_67248670d592.slice/crio-926f288e0a097658b3a2a480366b38dd94981db2a99c54376533d1dd6e4f40f4 WatchSource:0}: Error finding container 926f288e0a097658b3a2a480366b38dd94981db2a99c54376533d1dd6e4f40f4: Status 404 returned error can't find the container with id 926f288e0a097658b3a2a480366b38dd94981db2a99c54376533d1dd6e4f40f4 Nov 02 17:51:06.443059 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.443042 4310 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:51:06.445314 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.445296 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwgd\" (UniqueName: \"kubernetes.io/projected/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-kube-api-access-zvwgd\") pod \"maas-api-revoke-sub-04f2db06-rgc9k\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:06.587958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.587881 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:06.601058 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.601029 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:06.925591 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:06.925559 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k"] Nov 02 17:51:06.926761 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:06.926737 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1e6e49_7e35_40b7_aa82_75b609ea5eba.slice/crio-87cfea712678fcb2cb915d016ec8e1c768a74e1f56ff79a60967986514a783a9 WatchSource:0}: Error finding container 87cfea712678fcb2cb915d016ec8e1c768a74e1f56ff79a60967986514a783a9: Status 404 returned error can't find the container with id 87cfea712678fcb2cb915d016ec8e1c768a74e1f56ff79a60967986514a783a9 Nov 02 17:51:07.207546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:07.207446 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" event={"ID":"1f1e6e49-7e35-40b7-aa82-75b609ea5eba","Type":"ContainerStarted","Data":"87cfea712678fcb2cb915d016ec8e1c768a74e1f56ff79a60967986514a783a9"} Nov 02 17:51:07.208963 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:07.208930 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" event={"ID":"43996d70-c3d6-453f-bc7a-67248670d592","Type":"ContainerStarted","Data":"926f288e0a097658b3a2a480366b38dd94981db2a99c54376533d1dd6e4f40f4"} Nov 02 17:51:07.476543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:07.476497 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:08.026143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.026107 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7"] Nov 02 17:51:08.030348 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.030325 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:08.037501 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.037471 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7"] Nov 02 17:51:08.160356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.160318 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbpl7\" (UniqueName: \"kubernetes.io/projected/9e5472ba-0efd-47e8-921e-8906e56d4155-kube-api-access-qbpl7\") pod \"maas-api-revoke-sub-20638c17-nh2q7\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:08.160573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.160451 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20638c17-nh2q7\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:08.160649 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.160615 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20638c17-nh2q7\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:08.261655 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.261619 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbpl7\" (UniqueName: \"kubernetes.io/projected/9e5472ba-0efd-47e8-921e-8906e56d4155-kube-api-access-qbpl7\") pod \"maas-api-revoke-sub-20638c17-nh2q7\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:08.262173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.261685 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20638c17-nh2q7\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:08.262173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.261765 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20638c17-nh2q7\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:08.262383 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.262361 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20638c17-nh2q7\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:08.264386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.264364 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20638c17-nh2q7\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:08.270314 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.270285 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbpl7\" (UniqueName: \"kubernetes.io/projected/9e5472ba-0efd-47e8-921e-8906e56d4155-kube-api-access-qbpl7\") pod \"maas-api-revoke-sub-20638c17-nh2q7\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:08.358349 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.358227 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:08.383134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.383017 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:08.499595 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.499557 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6"] Nov 02 17:51:08.514044 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.514014 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:08.517786 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.517757 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6"] Nov 02 17:51:08.663037 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.663005 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7"] Nov 02 17:51:08.663820 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:08.663789 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5472ba_0efd_47e8_921e_8906e56d4155.slice/crio-db0621d5b07863003c8691b1a75a22bfe5c5c7d4ad6ff9a1449786a816bc3c94 WatchSource:0}: Error finding container db0621d5b07863003c8691b1a75a22bfe5c5c7d4ad6ff9a1449786a816bc3c94: Status 404 returned error can't find the container with id db0621d5b07863003c8691b1a75a22bfe5c5c7d4ad6ff9a1449786a816bc3c94 Nov 02 17:51:08.667949 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.667375 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a098faa-nkdx6\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:08.667949 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.667477 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a098faa-nkdx6\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:08.667949 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.667511 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jmpv\" (UniqueName: \"kubernetes.io/projected/dc447024-dc18-409b-af2a-de23bb927c21-kube-api-access-5jmpv\") pod \"maas-api-revoke-sub-3a098faa-nkdx6\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:08.768676 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.768637 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a098faa-nkdx6\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:08.768880 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.768772 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a098faa-nkdx6\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:08.768880 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.768795 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jmpv\" (UniqueName: \"kubernetes.io/projected/dc447024-dc18-409b-af2a-de23bb927c21-kube-api-access-5jmpv\") pod \"maas-api-revoke-sub-3a098faa-nkdx6\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:08.771382 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.771280 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a098faa-nkdx6\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:08.773445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.773417 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a098faa-nkdx6\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:08.779961 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.779937 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jmpv\" (UniqueName: \"kubernetes.io/projected/dc447024-dc18-409b-af2a-de23bb927c21-kube-api-access-5jmpv\") pod \"maas-api-revoke-sub-3a098faa-nkdx6\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:08.832961 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:08.832720 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:09.129514 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:09.129397 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6"] Nov 02 17:51:09.134330 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:09.134305 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc447024_dc18_409b_af2a_de23bb927c21.slice/crio-a16fde957f3fe19ed868ca390416e1db03c30746f7cd54162f2b101d7f5b71b9 WatchSource:0}: Error finding container a16fde957f3fe19ed868ca390416e1db03c30746f7cd54162f2b101d7f5b71b9: Status 404 returned error can't find the container with id a16fde957f3fe19ed868ca390416e1db03c30746f7cd54162f2b101d7f5b71b9 Nov 02 17:51:09.228246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:09.228207 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" event={"ID":"9e5472ba-0efd-47e8-921e-8906e56d4155","Type":"ContainerStarted","Data":"db0621d5b07863003c8691b1a75a22bfe5c5c7d4ad6ff9a1449786a816bc3c94"} Nov 02 17:51:09.229866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:09.229824 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" event={"ID":"dc447024-dc18-409b-af2a-de23bb927c21","Type":"ContainerStarted","Data":"a16fde957f3fe19ed868ca390416e1db03c30746f7cd54162f2b101d7f5b71b9"} Nov 02 17:51:09.247963 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:09.247930 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:09.993016 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:09.992958 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:10.239996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:10.239963 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" event={"ID":"9e5472ba-0efd-47e8-921e-8906e56d4155","Type":"ContainerStarted","Data":"e2411c4012ca2531fabc393b8d0f26aa2fa367f1e5af2740e7344c21cdc345aa"} Nov 02 17:51:10.241482 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:10.241452 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" event={"ID":"43996d70-c3d6-453f-bc7a-67248670d592","Type":"ContainerStarted","Data":"397d887daf3fa5fa1c97d3bf65b5bae48be942b49a35dcc87e47d636b0b53d01"} Nov 02 17:51:10.259816 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:10.259767 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" podStartSLOduration=1.361847785 podStartE2EDuration="2.259753463s" totalImagesPullingTime="897.905678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:10.258228159 +0000 UTC m=+667.701310797" watchObservedRunningTime="2026-11-02 17:51:10.259753463 +0000 UTC m=+667.702836099" Nov 02 17:51:10.278683 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:10.278635 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" podStartSLOduration=2.2548300980000002 podStartE2EDuration="5.278620907s" totalImagesPullingTime="3.023790809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:10.275070642 +0000 UTC m=+667.718153303" watchObservedRunningTime="2026-11-02 17:51:10.278620907 +0000 UTC m=+667.721703563" Nov 02 17:51:10.781554 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:10.781521 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:11.249307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:11.249270 4310 generic.go:356] "Generic (PLEG): container finished" podID="9e5472ba-0efd-47e8-921e-8906e56d4155" containerID="e2411c4012ca2531fabc393b8d0f26aa2fa367f1e5af2740e7344c21cdc345aa" exitCode=0 Nov 02 17:51:11.249758 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:11.249356 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" event={"ID":"9e5472ba-0efd-47e8-921e-8906e56d4155","Type":"ContainerDied","Data":"e2411c4012ca2531fabc393b8d0f26aa2fa367f1e5af2740e7344c21cdc345aa"} Nov 02 17:51:11.251154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:11.251123 4310 generic.go:356] "Generic (PLEG): container finished" podID="1f1e6e49-7e35-40b7-aa82-75b609ea5eba" containerID="2c66aab87c04bbfc22b42767e2436e7181d080845b7690d06f38a101bd6e51a7" exitCode=0 Nov 02 17:51:11.251277 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:11.251158 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" event={"ID":"1f1e6e49-7e35-40b7-aa82-75b609ea5eba","Type":"ContainerDied","Data":"2c66aab87c04bbfc22b42767e2436e7181d080845b7690d06f38a101bd6e51a7"} Nov 02 17:51:11.252849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:11.252830 4310 generic.go:356] "Generic (PLEG): container finished" podID="dc447024-dc18-409b-af2a-de23bb927c21" containerID="7176bad6074bb303b5241d6c28871543a70e0c827ca029442b02aebfb41478bc" exitCode=0 Nov 02 17:51:11.252961 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:11.252914 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" event={"ID":"dc447024-dc18-409b-af2a-de23bb927c21","Type":"ContainerDied","Data":"7176bad6074bb303b5241d6c28871543a70e0c827ca029442b02aebfb41478bc"} Nov 02 17:51:11.254545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:11.254527 4310 generic.go:356] "Generic (PLEG): container finished" podID="43996d70-c3d6-453f-bc7a-67248670d592" containerID="397d887daf3fa5fa1c97d3bf65b5bae48be942b49a35dcc87e47d636b0b53d01" exitCode=0 Nov 02 17:51:11.254625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:11.254573 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" event={"ID":"43996d70-c3d6-453f-bc7a-67248670d592","Type":"ContainerDied","Data":"397d887daf3fa5fa1c97d3bf65b5bae48be942b49a35dcc87e47d636b0b53d01"} Nov 02 17:51:11.464146 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:11.464117 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:12.263555 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.263511 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:12.449880 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.449481 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:12.485015 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.484984 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:12.521338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.521209 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5472ba-0efd-47e8-921e-8906e56d4155-kube-api-access-qbpl7\" (UniqueName: \"kubernetes.io/projected/9e5472ba-0efd-47e8-921e-8906e56d4155-kube-api-access-qbpl7\") pod \"9e5472ba-0efd-47e8-921e-8906e56d4155\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " Nov 02 17:51:12.521338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.521296 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-xks-ca\") pod \"9e5472ba-0efd-47e8-921e-8906e56d4155\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " Nov 02 17:51:12.522646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.521383 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-openshift-ca\") pod \"9e5472ba-0efd-47e8-921e-8906e56d4155\" (UID: \"9e5472ba-0efd-47e8-921e-8906e56d4155\") " Nov 02 17:51:12.522646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.522141 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-openshift-ca" pod "9e5472ba-0efd-47e8-921e-8906e56d4155" (UID: "9e5472ba-0efd-47e8-921e-8906e56d4155"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:12.524460 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.524427 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5472ba-0efd-47e8-921e-8906e56d4155-kube-api-access-qbpl7" pod "9e5472ba-0efd-47e8-921e-8906e56d4155" (UID: "9e5472ba-0efd-47e8-921e-8906e56d4155"). InnerVolumeSpecName "kube-api-access-qbpl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:12.527565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.527524 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-xks-ca" pod "9e5472ba-0efd-47e8-921e-8906e56d4155" (UID: "9e5472ba-0efd-47e8-921e-8906e56d4155"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:12.550819 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.550799 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:12.554327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.554304 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:12.622985 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.622950 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-openshift-ca\") pod \"dc447024-dc18-409b-af2a-de23bb927c21\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " Nov 02 17:51:12.622985 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.622990 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-xks-ca\") pod \"43996d70-c3d6-453f-bc7a-67248670d592\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " Nov 02 17:51:12.623262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623034 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-openshift-ca\") pod \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " Nov 02 17:51:12.623262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623092 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc447024-dc18-409b-af2a-de23bb927c21-kube-api-access-5jmpv\" (UniqueName: \"kubernetes.io/projected/dc447024-dc18-409b-af2a-de23bb927c21-kube-api-access-5jmpv\") pod \"dc447024-dc18-409b-af2a-de23bb927c21\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " Nov 02 17:51:12.623262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623114 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-xks-ca\") pod \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " Nov 02 17:51:12.623262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623162 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43996d70-c3d6-453f-bc7a-67248670d592-kube-api-access-cb6dj\" (UniqueName: \"kubernetes.io/projected/43996d70-c3d6-453f-bc7a-67248670d592-kube-api-access-cb6dj\") pod \"43996d70-c3d6-453f-bc7a-67248670d592\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " Nov 02 17:51:12.623262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623181 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-xks-ca\") pod \"dc447024-dc18-409b-af2a-de23bb927c21\" (UID: \"dc447024-dc18-409b-af2a-de23bb927c21\") " Nov 02 17:51:12.623262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623212 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-kube-api-access-zvwgd\" (UniqueName: \"kubernetes.io/projected/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-kube-api-access-zvwgd\") pod \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\" (UID: \"1f1e6e49-7e35-40b7-aa82-75b609ea5eba\") " Nov 02 17:51:12.623571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623267 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-openshift-ca\") pod \"43996d70-c3d6-453f-bc7a-67248670d592\" (UID: \"43996d70-c3d6-453f-bc7a-67248670d592\") " Nov 02 17:51:12.623571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623433 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-openshift-ca" pod "dc447024-dc18-409b-af2a-de23bb927c21" (UID: "dc447024-dc18-409b-af2a-de23bb927c21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:12.623571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623452 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-openshift-ca" pod "1f1e6e49-7e35-40b7-aa82-75b609ea5eba" (UID: "1f1e6e49-7e35-40b7-aa82-75b609ea5eba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:12.623837 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623815 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.623920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623846 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.623920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623861 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.623920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623874 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbpl7\" (UniqueName: \"kubernetes.io/projected/9e5472ba-0efd-47e8-921e-8906e56d4155-kube-api-access-qbpl7\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.623920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623889 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5472ba-0efd-47e8-921e-8906e56d4155-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.623920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.623820 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-openshift-ca" pod "43996d70-c3d6-453f-bc7a-67248670d592" (UID: "43996d70-c3d6-453f-bc7a-67248670d592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:12.625863 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.625824 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc447024-dc18-409b-af2a-de23bb927c21-kube-api-access-5jmpv" pod "dc447024-dc18-409b-af2a-de23bb927c21" (UID: "dc447024-dc18-409b-af2a-de23bb927c21"). InnerVolumeSpecName "kube-api-access-5jmpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:12.625962 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.625911 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43996d70-c3d6-453f-bc7a-67248670d592-kube-api-access-cb6dj" pod "43996d70-c3d6-453f-bc7a-67248670d592" (UID: "43996d70-c3d6-453f-bc7a-67248670d592"). InnerVolumeSpecName "kube-api-access-cb6dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:12.626053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.626035 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-xks-ca" pod "1f1e6e49-7e35-40b7-aa82-75b609ea5eba" (UID: "1f1e6e49-7e35-40b7-aa82-75b609ea5eba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:12.626663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.626628 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-kube-api-access-zvwgd" pod "1f1e6e49-7e35-40b7-aa82-75b609ea5eba" (UID: "1f1e6e49-7e35-40b7-aa82-75b609ea5eba"). InnerVolumeSpecName "kube-api-access-zvwgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:12.626799 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.626760 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-xks-ca" pod "43996d70-c3d6-453f-bc7a-67248670d592" (UID: "43996d70-c3d6-453f-bc7a-67248670d592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:12.627006 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.626985 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-xks-ca" pod "dc447024-dc18-409b-af2a-de23bb927c21" (UID: "dc447024-dc18-409b-af2a-de23bb927c21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:12.724632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.724602 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.724632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.724628 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43996d70-c3d6-453f-bc7a-67248670d592-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.724632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.724637 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jmpv\" (UniqueName: \"kubernetes.io/projected/dc447024-dc18-409b-af2a-de23bb927c21-kube-api-access-5jmpv\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.724839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.724646 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.724839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.724656 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb6dj\" (UniqueName: \"kubernetes.io/projected/43996d70-c3d6-453f-bc7a-67248670d592-kube-api-access-cb6dj\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.724839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.724666 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc447024-dc18-409b-af2a-de23bb927c21-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.724839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.724675 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvwgd\" (UniqueName: \"kubernetes.io/projected/1f1e6e49-7e35-40b7-aa82-75b609ea5eba-kube-api-access-zvwgd\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:12.975659 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:12.975619 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:13.287347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.287260 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" Nov 02 17:51:13.287347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.287289 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7" event={"ID":"9e5472ba-0efd-47e8-921e-8906e56d4155","Type":"ContainerDied","Data":"db0621d5b07863003c8691b1a75a22bfe5c5c7d4ad6ff9a1449786a816bc3c94"} Nov 02 17:51:13.287347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.287342 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db0621d5b07863003c8691b1a75a22bfe5c5c7d4ad6ff9a1449786a816bc3c94" Nov 02 17:51:13.288805 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.288785 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" Nov 02 17:51:13.288805 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.288794 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k" event={"ID":"1f1e6e49-7e35-40b7-aa82-75b609ea5eba","Type":"ContainerDied","Data":"87cfea712678fcb2cb915d016ec8e1c768a74e1f56ff79a60967986514a783a9"} Nov 02 17:51:13.288991 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.288839 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87cfea712678fcb2cb915d016ec8e1c768a74e1f56ff79a60967986514a783a9" Nov 02 17:51:13.290356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.290337 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" Nov 02 17:51:13.290466 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.290342 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6" event={"ID":"dc447024-dc18-409b-af2a-de23bb927c21","Type":"ContainerDied","Data":"a16fde957f3fe19ed868ca390416e1db03c30746f7cd54162f2b101d7f5b71b9"} Nov 02 17:51:13.290515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.290462 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a16fde957f3fe19ed868ca390416e1db03c30746f7cd54162f2b101d7f5b71b9" Nov 02 17:51:13.292321 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.292291 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" event={"ID":"43996d70-c3d6-453f-bc7a-67248670d592","Type":"ContainerDied","Data":"926f288e0a097658b3a2a480366b38dd94981db2a99c54376533d1dd6e4f40f4"} Nov 02 17:51:13.292414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.292365 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="926f288e0a097658b3a2a480366b38dd94981db2a99c54376533d1dd6e4f40f4" Nov 02 17:51:13.292414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.292301 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx" Nov 02 17:51:13.662255 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:13.662158 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:14.352613 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:14.352572 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:15.350119 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:15.347056 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:16.167386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:16.167350 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:16.906125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:16.906067 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:17.686013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.685976 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm"] Nov 02 17:51:17.686759 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.686743 4310 cpu_manager.go:414] "containerMap: removing container" podUID="9e5472ba-0efd-47e8-921e-8906e56d4155" containerName="revoke-keys" Nov 02 17:51:17.686812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.686762 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5472ba-0efd-47e8-921e-8906e56d4155" containerName="revoke-keys" Nov 02 17:51:17.686812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.686788 4310 cpu_manager.go:414] "containerMap: removing container" podUID="43996d70-c3d6-453f-bc7a-67248670d592" containerName="revoke-keys" Nov 02 17:51:17.686812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.686796 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43996d70-c3d6-453f-bc7a-67248670d592" containerName="revoke-keys" Nov 02 17:51:17.686911 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.686811 4310 cpu_manager.go:414] "containerMap: removing container" podUID="dc447024-dc18-409b-af2a-de23bb927c21" containerName="revoke-keys" Nov 02 17:51:17.686911 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.686820 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc447024-dc18-409b-af2a-de23bb927c21" containerName="revoke-keys" Nov 02 17:51:17.686911 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.686846 4310 cpu_manager.go:414] "containerMap: removing container" podUID="1f1e6e49-7e35-40b7-aa82-75b609ea5eba" containerName="revoke-keys" Nov 02 17:51:17.686911 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.686854 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1e6e49-7e35-40b7-aa82-75b609ea5eba" containerName="revoke-keys" Nov 02 17:51:17.687025 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.686973 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1e6e49-7e35-40b7-aa82-75b609ea5eba" containerName="revoke-keys" Nov 02 17:51:17.687025 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.686988 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5472ba-0efd-47e8-921e-8906e56d4155" containerName="revoke-keys" Nov 02 17:51:17.687025 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.686997 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="43996d70-c3d6-453f-bc7a-67248670d592" containerName="revoke-keys" Nov 02 17:51:17.687025 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.687006 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc447024-dc18-409b-af2a-de23bb927c21" containerName="revoke-keys" Nov 02 17:51:17.691858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.691835 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:17.703657 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.703632 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm"] Nov 02 17:51:17.705154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.705131 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:51:17.705433 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.705413 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:51:17.825928 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.824821 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc3589c-9dbmm\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:17.826482 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.826454 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmqch\" (UniqueName: \"kubernetes.io/projected/0b78e3cb-4752-4ac9-a3f9-066179814979-kube-api-access-tmqch\") pod \"maas-api-revoke-sub-3bc3589c-9dbmm\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:17.826700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.826681 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc3589c-9dbmm\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:17.893986 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.893957 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:17.921273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.921240 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg"] Nov 02 17:51:17.925276 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.925256 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:17.928157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.928135 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc3589c-9dbmm\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:17.928305 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.928288 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmqch\" (UniqueName: \"kubernetes.io/projected/0b78e3cb-4752-4ac9-a3f9-066179814979-kube-api-access-tmqch\") pod \"maas-api-revoke-sub-3bc3589c-9dbmm\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:17.928399 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.928381 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc3589c-9dbmm\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:17.929134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.929113 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc3589c-9dbmm\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:17.931879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.931853 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc3589c-9dbmm\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:17.934474 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.934449 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg"] Nov 02 17:51:17.940876 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:17.940813 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmqch\" (UniqueName: \"kubernetes.io/projected/0b78e3cb-4752-4ac9-a3f9-066179814979-kube-api-access-tmqch\") pod \"maas-api-revoke-sub-3bc3589c-9dbmm\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:18.007626 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.007584 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:18.029052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.029021 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af579afc-f5dzg\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:18.029180 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.029127 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af579afc-f5dzg\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:18.029180 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.029175 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdr42\" (UniqueName: \"kubernetes.io/projected/860782be-4a11-45ba-9515-ac67ae07091a-kube-api-access-xdr42\") pod \"maas-api-revoke-sub-af579afc-f5dzg\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:18.130878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.130839 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af579afc-f5dzg\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:18.131130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.130924 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af579afc-f5dzg\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:18.131130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.130995 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdr42\" (UniqueName: \"kubernetes.io/projected/860782be-4a11-45ba-9515-ac67ae07091a-kube-api-access-xdr42\") pod \"maas-api-revoke-sub-af579afc-f5dzg\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:18.131624 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.131598 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af579afc-f5dzg\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:18.133344 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.133325 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af579afc-f5dzg\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:18.139682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.139659 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdr42\" (UniqueName: \"kubernetes.io/projected/860782be-4a11-45ba-9515-ac67ae07091a-kube-api-access-xdr42\") pod \"maas-api-revoke-sub-af579afc-f5dzg\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:18.275560 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.275530 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:18.286227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.286203 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm"] Nov 02 17:51:18.288542 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:18.288511 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b78e3cb_4752_4ac9_a3f9_066179814979.slice/crio-c80e927c3002919a58a35c4e833d68ec4b374f4f8da0da14bbf206b6c4295321 WatchSource:0}: Error finding container c80e927c3002919a58a35c4e833d68ec4b374f4f8da0da14bbf206b6c4295321: Status 404 returned error can't find the container with id c80e927c3002919a58a35c4e833d68ec4b374f4f8da0da14bbf206b6c4295321 Nov 02 17:51:18.328638 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.328599 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" event={"ID":"0b78e3cb-4752-4ac9-a3f9-066179814979","Type":"ContainerStarted","Data":"c80e927c3002919a58a35c4e833d68ec4b374f4f8da0da14bbf206b6c4295321"} Nov 02 17:51:18.560958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.560931 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg"] Nov 02 17:51:18.563128 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:18.563055 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860782be_4a11_45ba_9515_ac67ae07091a.slice/crio-2684422a731c8e4d715c8fab5744ef061f04d2fd0237f2f307e61d6346d62a1a WatchSource:0}: Error finding container 2684422a731c8e4d715c8fab5744ef061f04d2fd0237f2f307e61d6346d62a1a: Status 404 returned error can't find the container with id 2684422a731c8e4d715c8fab5744ef061f04d2fd0237f2f307e61d6346d62a1a Nov 02 17:51:18.588299 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:18.588276 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:19.304473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:19.304426 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:19.352699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:19.352659 4310 generic.go:356] "Generic (PLEG): container finished" podID="0b78e3cb-4752-4ac9-a3f9-066179814979" containerID="82022934e8bc8293cb026539743ca42c789c9cfb6cb47ce4ce30176f67525c13" exitCode=0 Nov 02 17:51:19.352915 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:19.352890 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" event={"ID":"0b78e3cb-4752-4ac9-a3f9-066179814979","Type":"ContainerDied","Data":"82022934e8bc8293cb026539743ca42c789c9cfb6cb47ce4ce30176f67525c13"} Nov 02 17:51:19.355464 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:19.355432 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" event={"ID":"860782be-4a11-45ba-9515-ac67ae07091a","Type":"ContainerStarted","Data":"8a03c43841de91c5cd1edcfe0d063d54c5ec2fff684f3a3393ae5c5da4a5c40a"} Nov 02 17:51:19.355738 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:19.355718 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" event={"ID":"860782be-4a11-45ba-9515-ac67ae07091a","Type":"ContainerStarted","Data":"2684422a731c8e4d715c8fab5744ef061f04d2fd0237f2f307e61d6346d62a1a"} Nov 02 17:51:19.389218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:19.389168 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" podStartSLOduration=2.389154208 podStartE2EDuration="2.389154208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:19.386571936 +0000 UTC m=+676.829654574" watchObservedRunningTime="2026-11-02 17:51:19.389154208 +0000 UTC m=+676.832236843" Nov 02 17:51:20.067492 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.067455 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:20.365763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.365677 4310 generic.go:356] "Generic (PLEG): container finished" podID="860782be-4a11-45ba-9515-ac67ae07091a" containerID="8a03c43841de91c5cd1edcfe0d063d54c5ec2fff684f3a3393ae5c5da4a5c40a" exitCode=0 Nov 02 17:51:20.366156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.365767 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" event={"ID":"860782be-4a11-45ba-9515-ac67ae07091a","Type":"ContainerDied","Data":"8a03c43841de91c5cd1edcfe0d063d54c5ec2fff684f3a3393ae5c5da4a5c40a"} Nov 02 17:51:20.520587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.520560 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:20.562450 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.562418 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-xks-ca\") pod \"0b78e3cb-4752-4ac9-a3f9-066179814979\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " Nov 02 17:51:20.562638 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.562506 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-openshift-ca\") pod \"0b78e3cb-4752-4ac9-a3f9-066179814979\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " Nov 02 17:51:20.562698 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.562683 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b78e3cb-4752-4ac9-a3f9-066179814979-kube-api-access-tmqch\" (UniqueName: \"kubernetes.io/projected/0b78e3cb-4752-4ac9-a3f9-066179814979-kube-api-access-tmqch\") pod \"0b78e3cb-4752-4ac9-a3f9-066179814979\" (UID: \"0b78e3cb-4752-4ac9-a3f9-066179814979\") " Nov 02 17:51:20.562955 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.562927 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-openshift-ca" pod "0b78e3cb-4752-4ac9-a3f9-066179814979" (UID: "0b78e3cb-4752-4ac9-a3f9-066179814979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:20.563070 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.563045 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:20.564533 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.564507 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-xks-ca" pod "0b78e3cb-4752-4ac9-a3f9-066179814979" (UID: "0b78e3cb-4752-4ac9-a3f9-066179814979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:20.564733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.564719 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b78e3cb-4752-4ac9-a3f9-066179814979-kube-api-access-tmqch" pod "0b78e3cb-4752-4ac9-a3f9-066179814979" (UID: "0b78e3cb-4752-4ac9-a3f9-066179814979"). InnerVolumeSpecName "kube-api-access-tmqch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:20.664003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.663919 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmqch\" (UniqueName: \"kubernetes.io/projected/0b78e3cb-4752-4ac9-a3f9-066179814979-kube-api-access-tmqch\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:20.664003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.663948 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b78e3cb-4752-4ac9-a3f9-066179814979-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:20.858628 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:20.858593 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:21.374396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.374362 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" Nov 02 17:51:21.374396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.374385 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm" event={"ID":"0b78e3cb-4752-4ac9-a3f9-066179814979","Type":"ContainerDied","Data":"c80e927c3002919a58a35c4e833d68ec4b374f4f8da0da14bbf206b6c4295321"} Nov 02 17:51:21.374884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.374429 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c80e927c3002919a58a35c4e833d68ec4b374f4f8da0da14bbf206b6c4295321" Nov 02 17:51:21.529385 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.529359 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:21.571709 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.571680 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-xks-ca\") pod \"860782be-4a11-45ba-9515-ac67ae07091a\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " Nov 02 17:51:21.571882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.571751 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860782be-4a11-45ba-9515-ac67ae07091a-kube-api-access-xdr42\" (UniqueName: \"kubernetes.io/projected/860782be-4a11-45ba-9515-ac67ae07091a-kube-api-access-xdr42\") pod \"860782be-4a11-45ba-9515-ac67ae07091a\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " Nov 02 17:51:21.571882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.571838 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-openshift-ca\") pod \"860782be-4a11-45ba-9515-ac67ae07091a\" (UID: \"860782be-4a11-45ba-9515-ac67ae07091a\") " Nov 02 17:51:21.572271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.572238 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-openshift-ca" pod "860782be-4a11-45ba-9515-ac67ae07091a" (UID: "860782be-4a11-45ba-9515-ac67ae07091a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:21.572416 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.572397 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:21.574749 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.574725 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860782be-4a11-45ba-9515-ac67ae07091a-kube-api-access-xdr42" pod "860782be-4a11-45ba-9515-ac67ae07091a" (UID: "860782be-4a11-45ba-9515-ac67ae07091a"). InnerVolumeSpecName "kube-api-access-xdr42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:21.574882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.574830 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-xks-ca" pod "860782be-4a11-45ba-9515-ac67ae07091a" (UID: "860782be-4a11-45ba-9515-ac67ae07091a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:21.639202 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.639103 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:21.673492 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.673456 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860782be-4a11-45ba-9515-ac67ae07091a-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:21.673492 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:21.673482 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdr42\" (UniqueName: \"kubernetes.io/projected/860782be-4a11-45ba-9515-ac67ae07091a-kube-api-access-xdr42\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:22.042842 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.042799 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw"] Nov 02 17:51:22.044057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.044028 4310 cpu_manager.go:414] "containerMap: removing container" podUID="0b78e3cb-4752-4ac9-a3f9-066179814979" containerName="revoke-keys" Nov 02 17:51:22.044201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.044066 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b78e3cb-4752-4ac9-a3f9-066179814979" containerName="revoke-keys" Nov 02 17:51:22.044201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.044158 4310 cpu_manager.go:414] "containerMap: removing container" podUID="860782be-4a11-45ba-9515-ac67ae07091a" containerName="revoke-keys" Nov 02 17:51:22.044201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.044170 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860782be-4a11-45ba-9515-ac67ae07091a" containerName="revoke-keys" Nov 02 17:51:22.044477 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.044461 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b78e3cb-4752-4ac9-a3f9-066179814979" containerName="revoke-keys" Nov 02 17:51:22.044537 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.044495 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="860782be-4a11-45ba-9515-ac67ae07091a" containerName="revoke-keys" Nov 02 17:51:22.051067 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.051042 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:22.055821 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.055789 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw"] Nov 02 17:51:22.076991 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.076960 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhn8b\" (UniqueName: \"kubernetes.io/projected/28aaacdf-f1cf-407f-92f8-eba32640d71f-kube-api-access-dhn8b\") pod \"maas-api-revoke-sub-a415d5cf-6btnw\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:22.077140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.076996 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a415d5cf-6btnw\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:22.077140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.077117 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a415d5cf-6btnw\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:22.178287 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.178242 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhn8b\" (UniqueName: \"kubernetes.io/projected/28aaacdf-f1cf-407f-92f8-eba32640d71f-kube-api-access-dhn8b\") pod \"maas-api-revoke-sub-a415d5cf-6btnw\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:22.178287 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.178292 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a415d5cf-6btnw\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:22.178513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.178359 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a415d5cf-6btnw\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:22.179102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.179064 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a415d5cf-6btnw\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:22.180804 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.180780 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a415d5cf-6btnw\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:22.191576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.191552 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhn8b\" (UniqueName: \"kubernetes.io/projected/28aaacdf-f1cf-407f-92f8-eba32640d71f-kube-api-access-dhn8b\") pod \"maas-api-revoke-sub-a415d5cf-6btnw\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:22.371462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.371374 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:22.374428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.374300 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:22.380990 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.380966 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" Nov 02 17:51:22.381140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.380993 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg" event={"ID":"860782be-4a11-45ba-9515-ac67ae07091a","Type":"ContainerDied","Data":"2684422a731c8e4d715c8fab5744ef061f04d2fd0237f2f307e61d6346d62a1a"} Nov 02 17:51:22.381140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.381049 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2684422a731c8e4d715c8fab5744ef061f04d2fd0237f2f307e61d6346d62a1a" Nov 02 17:51:22.674681 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:22.674636 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28aaacdf_f1cf_407f_92f8_eba32640d71f.slice/crio-bbd94cb9572d0b0a00ea46a3a153fb300a9e142b742cc3cc4a8c3474964d0f38 WatchSource:0}: Error finding container bbd94cb9572d0b0a00ea46a3a153fb300a9e142b742cc3cc4a8c3474964d0f38: Status 404 returned error can't find the container with id bbd94cb9572d0b0a00ea46a3a153fb300a9e142b742cc3cc4a8c3474964d0f38 Nov 02 17:51:22.675318 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:22.675296 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw"] Nov 02 17:51:23.154649 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:23.154616 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:23.389463 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:23.389419 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" event={"ID":"28aaacdf-f1cf-407f-92f8-eba32640d71f","Type":"ContainerStarted","Data":"356ba5e35aed923627526ddab7dcd226bce478e0bd4895153f324ef416b217e1"} Nov 02 17:51:23.389961 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:23.389484 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" event={"ID":"28aaacdf-f1cf-407f-92f8-eba32640d71f","Type":"ContainerStarted","Data":"bbd94cb9572d0b0a00ea46a3a153fb300a9e142b742cc3cc4a8c3474964d0f38"} Nov 02 17:51:23.420176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:23.420123 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" podStartSLOduration=1.4201071459999999 podStartE2EDuration="1.420107146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:23.417646827 +0000 UTC m=+680.860729478" watchObservedRunningTime="2026-11-02 17:51:23.420107146 +0000 UTC m=+680.863189790" Nov 02 17:51:24.251408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.251364 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv"] Nov 02 17:51:24.252632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.252592 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:24.255874 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.255853 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:24.270643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.270619 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv"] Nov 02 17:51:24.339564 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.338091 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4717896-vbrnv\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:24.339564 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.338141 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4717896-vbrnv\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:24.339564 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.338195 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdxb\" (UniqueName: \"kubernetes.io/projected/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-kube-api-access-nxdxb\") pod \"maas-api-revoke-sub-a4717896-vbrnv\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:24.399102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.398728 4310 generic.go:356] "Generic (PLEG): container finished" podID="28aaacdf-f1cf-407f-92f8-eba32640d71f" containerID="356ba5e35aed923627526ddab7dcd226bce478e0bd4895153f324ef416b217e1" exitCode=0 Nov 02 17:51:24.399102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.398891 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" event={"ID":"28aaacdf-f1cf-407f-92f8-eba32640d71f","Type":"ContainerDied","Data":"356ba5e35aed923627526ddab7dcd226bce478e0bd4895153f324ef416b217e1"} Nov 02 17:51:24.439789 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.439751 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4717896-vbrnv\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:24.439789 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.439800 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4717896-vbrnv\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:24.440041 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.439859 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdxb\" (UniqueName: \"kubernetes.io/projected/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-kube-api-access-nxdxb\") pod \"maas-api-revoke-sub-a4717896-vbrnv\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:24.440588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.440559 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4717896-vbrnv\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:24.442540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.442514 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4717896-vbrnv\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:24.449002 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.448969 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdxb\" (UniqueName: \"kubernetes.io/projected/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-kube-api-access-nxdxb\") pod \"maas-api-revoke-sub-a4717896-vbrnv\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:24.570591 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.570496 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:24.876103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.864427 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv"] Nov 02 17:51:24.889338 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:24.885234 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390b7a47_77c3_4a9f_8bc8_ed6735f823ac.slice/crio-309d6143b6baf3fc7034afc234b9397d873bd91fbb44a9a34b209dba883c8fef WatchSource:0}: Error finding container 309d6143b6baf3fc7034afc234b9397d873bd91fbb44a9a34b209dba883c8fef: Status 404 returned error can't find the container with id 309d6143b6baf3fc7034afc234b9397d873bd91fbb44a9a34b209dba883c8fef Nov 02 17:51:24.923149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.923127 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86"] Nov 02 17:51:24.928032 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.928012 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:24.936773 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.936743 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86"] Nov 02 17:51:24.949107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.946347 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd0b5bab-t7z86\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:24.949107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.946395 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd0b5bab-t7z86\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:24.949107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:24.946556 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f22bn\" (UniqueName: \"kubernetes.io/projected/77ee8219-2afe-469b-abaa-da3305f86cf2-kube-api-access-f22bn\") pod \"maas-api-revoke-sub-bd0b5bab-t7z86\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:25.048042 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.047994 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f22bn\" (UniqueName: \"kubernetes.io/projected/77ee8219-2afe-469b-abaa-da3305f86cf2-kube-api-access-f22bn\") pod \"maas-api-revoke-sub-bd0b5bab-t7z86\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:25.048216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.048127 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd0b5bab-t7z86\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:25.048216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.048181 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd0b5bab-t7z86\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:25.050658 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.050631 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd0b5bab-t7z86\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:25.053906 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.053880 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd0b5bab-t7z86\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:25.058794 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.058773 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f22bn\" (UniqueName: \"kubernetes.io/projected/77ee8219-2afe-469b-abaa-da3305f86cf2-kube-api-access-f22bn\") pod \"maas-api-revoke-sub-bd0b5bab-t7z86\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:25.084181 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.083749 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:25.308925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.308891 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:25.415675 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.415640 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" event={"ID":"390b7a47-77c3-4a9f-8bc8-ed6735f823ac","Type":"ContainerStarted","Data":"d81b2d1481a6809245086a5c78a830f7520b990722c9b5fd09664b74b51c3942"} Nov 02 17:51:25.416012 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.415697 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" event={"ID":"390b7a47-77c3-4a9f-8bc8-ed6735f823ac","Type":"ContainerStarted","Data":"309d6143b6baf3fc7034afc234b9397d873bd91fbb44a9a34b209dba883c8fef"} Nov 02 17:51:25.433101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.431432 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" podStartSLOduration=1.431413807 podStartE2EDuration="1.431413807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:25.430735924 +0000 UTC m=+682.873818564" watchObservedRunningTime="2026-11-02 17:51:25.431413807 +0000 UTC m=+682.874496445" Nov 02 17:51:25.637052 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:25.637020 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ee8219_2afe_469b_abaa_da3305f86cf2.slice/crio-281e3d6c7000b61a4c8db5a77d3c37eeb14d2b03923fc9bf1bfb484f837c396b WatchSource:0}: Error finding container 281e3d6c7000b61a4c8db5a77d3c37eeb14d2b03923fc9bf1bfb484f837c396b: Status 404 returned error can't find the container with id 281e3d6c7000b61a4c8db5a77d3c37eeb14d2b03923fc9bf1bfb484f837c396b Nov 02 17:51:25.638837 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.638806 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86"] Nov 02 17:51:25.810877 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.810527 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:25.871779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.871746 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:25.925226 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.925193 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28aaacdf-f1cf-407f-92f8-eba32640d71f-kube-api-access-dhn8b\" (UniqueName: \"kubernetes.io/projected/28aaacdf-f1cf-407f-92f8-eba32640d71f-kube-api-access-dhn8b\") pod \"28aaacdf-f1cf-407f-92f8-eba32640d71f\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " Nov 02 17:51:25.925740 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.925711 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-xks-ca\") pod \"28aaacdf-f1cf-407f-92f8-eba32640d71f\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " Nov 02 17:51:25.926144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.925920 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-openshift-ca\") pod \"28aaacdf-f1cf-407f-92f8-eba32640d71f\" (UID: \"28aaacdf-f1cf-407f-92f8-eba32640d71f\") " Nov 02 17:51:25.927528 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.927318 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-openshift-ca" pod "28aaacdf-f1cf-407f-92f8-eba32640d71f" (UID: "28aaacdf-f1cf-407f-92f8-eba32640d71f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:25.940102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.937177 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28aaacdf-f1cf-407f-92f8-eba32640d71f-kube-api-access-dhn8b" pod "28aaacdf-f1cf-407f-92f8-eba32640d71f" (UID: "28aaacdf-f1cf-407f-92f8-eba32640d71f"). InnerVolumeSpecName "kube-api-access-dhn8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:25.940102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:25.939039 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-xks-ca" pod "28aaacdf-f1cf-407f-92f8-eba32640d71f" (UID: "28aaacdf-f1cf-407f-92f8-eba32640d71f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:26.027809 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.027772 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:26.027809 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.027803 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhn8b\" (UniqueName: \"kubernetes.io/projected/28aaacdf-f1cf-407f-92f8-eba32640d71f-kube-api-access-dhn8b\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:26.027809 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.027815 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28aaacdf-f1cf-407f-92f8-eba32640d71f-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:26.426465 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.426374 4310 generic.go:356] "Generic (PLEG): container finished" podID="390b7a47-77c3-4a9f-8bc8-ed6735f823ac" containerID="d81b2d1481a6809245086a5c78a830f7520b990722c9b5fd09664b74b51c3942" exitCode=0 Nov 02 17:51:26.426853 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.426466 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" event={"ID":"390b7a47-77c3-4a9f-8bc8-ed6735f823ac","Type":"ContainerDied","Data":"d81b2d1481a6809245086a5c78a830f7520b990722c9b5fd09664b74b51c3942"} Nov 02 17:51:26.427960 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.427936 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" Nov 02 17:51:26.428065 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.427981 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw" event={"ID":"28aaacdf-f1cf-407f-92f8-eba32640d71f","Type":"ContainerDied","Data":"bbd94cb9572d0b0a00ea46a3a153fb300a9e142b742cc3cc4a8c3474964d0f38"} Nov 02 17:51:26.428065 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.428032 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbd94cb9572d0b0a00ea46a3a153fb300a9e142b742cc3cc4a8c3474964d0f38" Nov 02 17:51:26.429618 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.429596 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" event={"ID":"77ee8219-2afe-469b-abaa-da3305f86cf2","Type":"ContainerStarted","Data":"dc4c33f892eefaf2fe587b695b35854f0cda290763560d8aa334d2e0dcb4342a"} Nov 02 17:51:26.429737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.429636 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" event={"ID":"77ee8219-2afe-469b-abaa-da3305f86cf2","Type":"ContainerStarted","Data":"281e3d6c7000b61a4c8db5a77d3c37eeb14d2b03923fc9bf1bfb484f837c396b"} Nov 02 17:51:26.481873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.481804 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" podStartSLOduration=2.481780417 podStartE2EDuration="2.481780417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:26.475461142 +0000 UTC m=+683.918543780" watchObservedRunningTime="2026-11-02 17:51:26.481780417 +0000 UTC m=+683.924863072" Nov 02 17:51:26.659893 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:26.659850 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:27.437427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.437395 4310 generic.go:356] "Generic (PLEG): container finished" podID="77ee8219-2afe-469b-abaa-da3305f86cf2" containerID="dc4c33f892eefaf2fe587b695b35854f0cda290763560d8aa334d2e0dcb4342a" exitCode=0 Nov 02 17:51:27.437879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.437483 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" event={"ID":"77ee8219-2afe-469b-abaa-da3305f86cf2","Type":"ContainerDied","Data":"dc4c33f892eefaf2fe587b695b35854f0cda290763560d8aa334d2e0dcb4342a"} Nov 02 17:51:27.528166 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.527937 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:27.614099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.614059 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:27.755840 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.755807 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-openshift-ca\") pod \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " Nov 02 17:51:27.756010 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.755899 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-xks-ca\") pod \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " Nov 02 17:51:27.756105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.756065 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-kube-api-access-nxdxb\" (UniqueName: \"kubernetes.io/projected/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-kube-api-access-nxdxb\") pod \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\" (UID: \"390b7a47-77c3-4a9f-8bc8-ed6735f823ac\") " Nov 02 17:51:27.756233 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.756207 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-openshift-ca" pod "390b7a47-77c3-4a9f-8bc8-ed6735f823ac" (UID: "390b7a47-77c3-4a9f-8bc8-ed6735f823ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:27.756537 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.756514 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:27.757984 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.757960 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-xks-ca" pod "390b7a47-77c3-4a9f-8bc8-ed6735f823ac" (UID: "390b7a47-77c3-4a9f-8bc8-ed6735f823ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:27.758099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.758050 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-kube-api-access-nxdxb" pod "390b7a47-77c3-4a9f-8bc8-ed6735f823ac" (UID: "390b7a47-77c3-4a9f-8bc8-ed6735f823ac"). InnerVolumeSpecName "kube-api-access-nxdxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:27.857393 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.857356 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:27.857393 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:27.857390 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxdxb\" (UniqueName: \"kubernetes.io/projected/390b7a47-77c3-4a9f-8bc8-ed6735f823ac-kube-api-access-nxdxb\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:28.359545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.359506 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:28.450830 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.450799 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" Nov 02 17:51:28.454761 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.454720 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv" event={"ID":"390b7a47-77c3-4a9f-8bc8-ed6735f823ac","Type":"ContainerDied","Data":"309d6143b6baf3fc7034afc234b9397d873bd91fbb44a9a34b209dba883c8fef"} Nov 02 17:51:28.455021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.454993 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="309d6143b6baf3fc7034afc234b9397d873bd91fbb44a9a34b209dba883c8fef" Nov 02 17:51:28.616003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.615928 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:28.666420 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.666379 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ee8219-2afe-469b-abaa-da3305f86cf2-kube-api-access-f22bn\" (UniqueName: \"kubernetes.io/projected/77ee8219-2afe-469b-abaa-da3305f86cf2-kube-api-access-f22bn\") pod \"77ee8219-2afe-469b-abaa-da3305f86cf2\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " Nov 02 17:51:28.666593 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.666488 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-openshift-ca\") pod \"77ee8219-2afe-469b-abaa-da3305f86cf2\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " Nov 02 17:51:28.666652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.666597 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-xks-ca\") pod \"77ee8219-2afe-469b-abaa-da3305f86cf2\" (UID: \"77ee8219-2afe-469b-abaa-da3305f86cf2\") " Nov 02 17:51:28.668114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.667367 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-openshift-ca" pod "77ee8219-2afe-469b-abaa-da3305f86cf2" (UID: "77ee8219-2afe-469b-abaa-da3305f86cf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:28.669863 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.669748 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-xks-ca" pod "77ee8219-2afe-469b-abaa-da3305f86cf2" (UID: "77ee8219-2afe-469b-abaa-da3305f86cf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:28.679727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.679696 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ee8219-2afe-469b-abaa-da3305f86cf2-kube-api-access-f22bn" pod "77ee8219-2afe-469b-abaa-da3305f86cf2" (UID: "77ee8219-2afe-469b-abaa-da3305f86cf2"). InnerVolumeSpecName "kube-api-access-f22bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:28.768167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.768130 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:28.768167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.768165 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ee8219-2afe-469b-abaa-da3305f86cf2-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:28.768416 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:28.768179 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f22bn\" (UniqueName: \"kubernetes.io/projected/77ee8219-2afe-469b-abaa-da3305f86cf2-kube-api-access-f22bn\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:29.152688 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.152660 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:29.458550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.458467 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" event={"ID":"77ee8219-2afe-469b-abaa-da3305f86cf2","Type":"ContainerDied","Data":"281e3d6c7000b61a4c8db5a77d3c37eeb14d2b03923fc9bf1bfb484f837c396b"} Nov 02 17:51:29.458550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.458518 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="281e3d6c7000b61a4c8db5a77d3c37eeb14d2b03923fc9bf1bfb484f837c396b" Nov 02 17:51:29.458955 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.458586 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86" Nov 02 17:51:29.670981 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.670948 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv"] Nov 02 17:51:29.671612 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.671592 4310 cpu_manager.go:414] "containerMap: removing container" podUID="390b7a47-77c3-4a9f-8bc8-ed6735f823ac" containerName="revoke-keys" Nov 02 17:51:29.671713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.671613 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390b7a47-77c3-4a9f-8bc8-ed6735f823ac" containerName="revoke-keys" Nov 02 17:51:29.671713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.671642 4310 cpu_manager.go:414] "containerMap: removing container" podUID="28aaacdf-f1cf-407f-92f8-eba32640d71f" containerName="revoke-keys" Nov 02 17:51:29.671713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.671650 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28aaacdf-f1cf-407f-92f8-eba32640d71f" containerName="revoke-keys" Nov 02 17:51:29.671713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.671662 4310 cpu_manager.go:414] "containerMap: removing container" podUID="77ee8219-2afe-469b-abaa-da3305f86cf2" containerName="revoke-keys" Nov 02 17:51:29.671713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.671670 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ee8219-2afe-469b-abaa-da3305f86cf2" containerName="revoke-keys" Nov 02 17:51:29.671965 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.671792 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="390b7a47-77c3-4a9f-8bc8-ed6735f823ac" containerName="revoke-keys" Nov 02 17:51:29.671965 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.671807 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="28aaacdf-f1cf-407f-92f8-eba32640d71f" containerName="revoke-keys" Nov 02 17:51:29.671965 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.671817 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ee8219-2afe-469b-abaa-da3305f86cf2" containerName="revoke-keys" Nov 02 17:51:29.684559 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.684531 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:29.693743 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.693419 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:51:29.694052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.693825 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:51:29.696114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.695295 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv"] Nov 02 17:51:29.778147 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.778117 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55ad14ed-cfjvv\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:29.778329 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.778159 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55ad14ed-cfjvv\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:29.778329 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.778247 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnv8\" (UniqueName: \"kubernetes.io/projected/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-kube-api-access-4tnv8\") pod \"maas-api-revoke-sub-55ad14ed-cfjvv\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:29.863691 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.863654 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:29.880063 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.880024 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55ad14ed-cfjvv\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:29.880475 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.880095 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55ad14ed-cfjvv\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:29.880475 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.880148 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnv8\" (UniqueName: \"kubernetes.io/projected/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-kube-api-access-4tnv8\") pod \"maas-api-revoke-sub-55ad14ed-cfjvv\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:29.881379 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.881355 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55ad14ed-cfjvv\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:29.883153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.883127 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55ad14ed-cfjvv\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:29.889692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.889669 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn"] Nov 02 17:51:29.894331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.894301 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:29.900509 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.899573 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnv8\" (UniqueName: \"kubernetes.io/projected/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-kube-api-access-4tnv8\") pod \"maas-api-revoke-sub-55ad14ed-cfjvv\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:29.911175 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.903660 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn"] Nov 02 17:51:29.981744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.981706 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rcz4\" (UniqueName: \"kubernetes.io/projected/266551f9-492e-4113-ac11-5f7fc6ba0304-kube-api-access-7rcz4\") pod \"maas-api-revoke-sub-2b5d5ac3-b2bvn\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:29.981892 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.981866 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5d5ac3-b2bvn\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:29.982041 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:29.982018 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5d5ac3-b2bvn\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:30.004611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.004571 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:30.083425 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.083389 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5d5ac3-b2bvn\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:30.083608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.083509 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5d5ac3-b2bvn\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:30.083673 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.083632 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rcz4\" (UniqueName: \"kubernetes.io/projected/266551f9-492e-4113-ac11-5f7fc6ba0304-kube-api-access-7rcz4\") pod \"maas-api-revoke-sub-2b5d5ac3-b2bvn\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:30.084241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.084216 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5d5ac3-b2bvn\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:30.085824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.085804 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5d5ac3-b2bvn\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:30.099649 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.099616 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rcz4\" (UniqueName: \"kubernetes.io/projected/266551f9-492e-4113-ac11-5f7fc6ba0304-kube-api-access-7rcz4\") pod \"maas-api-revoke-sub-2b5d5ac3-b2bvn\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:30.246003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.245964 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:30.286822 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.285977 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv"] Nov 02 17:51:30.288522 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:30.288185 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8466b8_2d52_4b4a_85a7_a566b8ad691d.slice/crio-06b2fc560aafd7dbd5c6fc8be505891036bd9be1ab401be5d7e11e1d8ca130a4 WatchSource:0}: Error finding container 06b2fc560aafd7dbd5c6fc8be505891036bd9be1ab401be5d7e11e1d8ca130a4: Status 404 returned error can't find the container with id 06b2fc560aafd7dbd5c6fc8be505891036bd9be1ab401be5d7e11e1d8ca130a4 Nov 02 17:51:30.466807 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.466766 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" event={"ID":"2a8466b8-2d52-4b4a-85a7-a566b8ad691d","Type":"ContainerStarted","Data":"06b2fc560aafd7dbd5c6fc8be505891036bd9be1ab401be5d7e11e1d8ca130a4"} Nov 02 17:51:30.529664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.529635 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn"] Nov 02 17:51:30.530176 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:30.530141 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266551f9_492e_4113_ac11_5f7fc6ba0304.slice/crio-40450020db90a1e53bbaf4dff9fef87ee131f6248a2cc23a005773cbaac8885c WatchSource:0}: Error finding container 40450020db90a1e53bbaf4dff9fef87ee131f6248a2cc23a005773cbaac8885c: Status 404 returned error can't find the container with id 40450020db90a1e53bbaf4dff9fef87ee131f6248a2cc23a005773cbaac8885c Nov 02 17:51:30.667491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:30.667450 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:31.018781 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.018746 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn"] Nov 02 17:51:31.022934 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.022917 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:31.030419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.030387 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn"] Nov 02 17:51:31.097859 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.097814 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0324a29-hdzdn\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:31.098041 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.097928 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bd66\" (UniqueName: \"kubernetes.io/projected/fb63e790-c7bc-4312-9afb-30049d3bb405-kube-api-access-6bd66\") pod \"maas-api-revoke-sub-f0324a29-hdzdn\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:31.098041 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.097997 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0324a29-hdzdn\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:31.199399 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.199366 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0324a29-hdzdn\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:31.199567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.199418 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bd66\" (UniqueName: \"kubernetes.io/projected/fb63e790-c7bc-4312-9afb-30049d3bb405-kube-api-access-6bd66\") pod \"maas-api-revoke-sub-f0324a29-hdzdn\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:31.199567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.199467 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0324a29-hdzdn\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:31.200177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.200159 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0324a29-hdzdn\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:31.201704 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.201686 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0324a29-hdzdn\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:31.206996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.206978 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bd66\" (UniqueName: \"kubernetes.io/projected/fb63e790-c7bc-4312-9afb-30049d3bb405-kube-api-access-6bd66\") pod \"maas-api-revoke-sub-f0324a29-hdzdn\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:31.343394 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.343356 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:31.474698 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.474668 4310 generic.go:356] "Generic (PLEG): container finished" podID="2a8466b8-2d52-4b4a-85a7-a566b8ad691d" containerID="fda5dd6ffee7d845402278e92b2ea8fa4348bde281144d091e9ea8f1c605586a" exitCode=0 Nov 02 17:51:31.475123 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.474755 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" event={"ID":"2a8466b8-2d52-4b4a-85a7-a566b8ad691d","Type":"ContainerDied","Data":"fda5dd6ffee7d845402278e92b2ea8fa4348bde281144d091e9ea8f1c605586a"} Nov 02 17:51:31.476453 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.476432 4310 generic.go:356] "Generic (PLEG): container finished" podID="266551f9-492e-4113-ac11-5f7fc6ba0304" containerID="64c178df3d753debe23fb7d8228960119f769d9699964df2c8f997e20ebbd8fc" exitCode=0 Nov 02 17:51:31.476544 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.476508 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" event={"ID":"266551f9-492e-4113-ac11-5f7fc6ba0304","Type":"ContainerDied","Data":"64c178df3d753debe23fb7d8228960119f769d9699964df2c8f997e20ebbd8fc"} Nov 02 17:51:31.476588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.476547 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" event={"ID":"266551f9-492e-4113-ac11-5f7fc6ba0304","Type":"ContainerStarted","Data":"40450020db90a1e53bbaf4dff9fef87ee131f6248a2cc23a005773cbaac8885c"} Nov 02 17:51:31.497441 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.497411 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:31.620768 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:31.620741 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn"] Nov 02 17:51:31.621789 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:31.621757 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb63e790_c7bc_4312_9afb_30049d3bb405.slice/crio-eb722760fd01990319616a9c09926e827554c3f35b4c3945d4480983761a01cb WatchSource:0}: Error finding container eb722760fd01990319616a9c09926e827554c3f35b4c3945d4480983761a01cb: Status 404 returned error can't find the container with id eb722760fd01990319616a9c09926e827554c3f35b4c3945d4480983761a01cb Nov 02 17:51:32.074491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.074460 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs"] Nov 02 17:51:32.078626 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.078602 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.090979 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.090821 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb4ecb\"/\"default-dockercfg-gjl4f\"" Nov 02 17:51:32.090979 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.090883 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb4ecb\"/\"openshift-service-ca.crt\"" Nov 02 17:51:32.090979 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.090947 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb4ecb\"/\"kube-root-ca.crt\"" Nov 02 17:51:32.091396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.091316 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb4ecb\"/\"distinct-w2-bb4ecb-kserve-self-signed-certs\"" Nov 02 17:51:32.094646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.094619 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs"] Nov 02 17:51:32.125141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.125053 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:32.210748 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.210712 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85ade69-b5f2-4498-a3de-2a1d8a084230-tls-certs\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.210926 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.210768 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-model-cache\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.210926 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.210871 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-home\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.211018 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.210987 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbh9\" (UniqueName: \"kubernetes.io/projected/d85ade69-b5f2-4498-a3de-2a1d8a084230-kube-api-access-rmbh9\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.211059 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.211038 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-tmp-dir\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.211148 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.211133 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-dshm\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.225500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.225466 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:32.326101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.322780 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85ade69-b5f2-4498-a3de-2a1d8a084230-tls-certs\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.326101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.322872 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-model-cache\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.326101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.322929 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-home\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.326101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.323039 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbh9\" (UniqueName: \"kubernetes.io/projected/d85ade69-b5f2-4498-a3de-2a1d8a084230-kube-api-access-rmbh9\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.326101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.323071 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-tmp-dir\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.326101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.323148 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-dshm\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.326101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.323549 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-model-cache\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.326101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.324476 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-home\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.326101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.324713 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-tmp-dir\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.355100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.352044 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-dshm\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.363364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.363245 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbh9\" (UniqueName: \"kubernetes.io/projected/d85ade69-b5f2-4498-a3de-2a1d8a084230-kube-api-access-rmbh9\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.364944 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.364886 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85ade69-b5f2-4498-a3de-2a1d8a084230-tls-certs\") pod \"distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.394798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.394724 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:32.486196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.486026 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" event={"ID":"fb63e790-c7bc-4312-9afb-30049d3bb405","Type":"ContainerStarted","Data":"a7aa8b5a1125c0ff17e6575affb256865f2ffdadb5caed2fcc1d4c812f499beb"} Nov 02 17:51:32.486196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.486112 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" event={"ID":"fb63e790-c7bc-4312-9afb-30049d3bb405","Type":"ContainerStarted","Data":"eb722760fd01990319616a9c09926e827554c3f35b4c3945d4480983761a01cb"} Nov 02 17:51:32.506106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.505722 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" podStartSLOduration=2.505704199 podStartE2EDuration="2.505704199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:32.5029553 +0000 UTC m=+689.946037940" watchObservedRunningTime="2026-11-02 17:51:32.505704199 +0000 UTC m=+689.948786834" Nov 02 17:51:32.681246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.681215 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs"] Nov 02 17:51:32.698998 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:32.698969 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd85ade69_b5f2_4498_a3de_2a1d8a084230.slice/crio-2ff162a322c0f4dce90accaf8fabba271954e731778d80f8b287023eea799f7d WatchSource:0}: Error finding container 2ff162a322c0f4dce90accaf8fabba271954e731778d80f8b287023eea799f7d: Status 404 returned error can't find the container with id 2ff162a322c0f4dce90accaf8fabba271954e731778d80f8b287023eea799f7d Nov 02 17:51:32.709266 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.709241 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:32.714870 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.714838 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:32.830394 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.830366 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266551f9-492e-4113-ac11-5f7fc6ba0304-kube-api-access-7rcz4\" (UniqueName: \"kubernetes.io/projected/266551f9-492e-4113-ac11-5f7fc6ba0304-kube-api-access-7rcz4\") pod \"266551f9-492e-4113-ac11-5f7fc6ba0304\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " Nov 02 17:51:32.830394 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.830406 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-openshift-ca\") pod \"266551f9-492e-4113-ac11-5f7fc6ba0304\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " Nov 02 17:51:32.830615 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.830595 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-xks-ca\") pod \"266551f9-492e-4113-ac11-5f7fc6ba0304\" (UID: \"266551f9-492e-4113-ac11-5f7fc6ba0304\") " Nov 02 17:51:32.830687 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.830657 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-xks-ca\") pod \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " Nov 02 17:51:32.830749 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.830713 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-openshift-ca\") pod \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " Nov 02 17:51:32.830801 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.830747 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-kube-api-access-4tnv8\" (UniqueName: \"kubernetes.io/projected/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-kube-api-access-4tnv8\") pod \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\" (UID: \"2a8466b8-2d52-4b4a-85a7-a566b8ad691d\") " Nov 02 17:51:32.830850 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.830796 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-openshift-ca" pod "266551f9-492e-4113-ac11-5f7fc6ba0304" (UID: "266551f9-492e-4113-ac11-5f7fc6ba0304"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:32.831166 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.831128 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-openshift-ca" pod "2a8466b8-2d52-4b4a-85a7-a566b8ad691d" (UID: "2a8466b8-2d52-4b4a-85a7-a566b8ad691d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:32.831362 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.831345 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:32.831448 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.831367 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:32.832891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.832861 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-xks-ca" pod "266551f9-492e-4113-ac11-5f7fc6ba0304" (UID: "266551f9-492e-4113-ac11-5f7fc6ba0304"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:32.833315 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.833293 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-xks-ca" pod "2a8466b8-2d52-4b4a-85a7-a566b8ad691d" (UID: "2a8466b8-2d52-4b4a-85a7-a566b8ad691d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:32.833364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.833339 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-kube-api-access-4tnv8" pod "2a8466b8-2d52-4b4a-85a7-a566b8ad691d" (UID: "2a8466b8-2d52-4b4a-85a7-a566b8ad691d"). InnerVolumeSpecName "kube-api-access-4tnv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:32.833416 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.833366 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266551f9-492e-4113-ac11-5f7fc6ba0304-kube-api-access-7rcz4" pod "266551f9-492e-4113-ac11-5f7fc6ba0304" (UID: "266551f9-492e-4113-ac11-5f7fc6ba0304"). InnerVolumeSpecName "kube-api-access-7rcz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:32.932441 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.932363 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tnv8\" (UniqueName: \"kubernetes.io/projected/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-kube-api-access-4tnv8\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:32.932441 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.932402 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rcz4\" (UniqueName: \"kubernetes.io/projected/266551f9-492e-4113-ac11-5f7fc6ba0304-kube-api-access-7rcz4\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:32.932441 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.932418 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/266551f9-492e-4113-ac11-5f7fc6ba0304-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:32.932441 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:32.932433 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8466b8-2d52-4b4a-85a7-a566b8ad691d-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:33.038901 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.038865 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:33.493948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.493904 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" event={"ID":"d85ade69-b5f2-4498-a3de-2a1d8a084230","Type":"ContainerStarted","Data":"db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f"} Nov 02 17:51:33.494440 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.493964 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" event={"ID":"d85ade69-b5f2-4498-a3de-2a1d8a084230","Type":"ContainerStarted","Data":"2ff162a322c0f4dce90accaf8fabba271954e731778d80f8b287023eea799f7d"} Nov 02 17:51:33.495389 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.495366 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" Nov 02 17:51:33.495527 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.495388 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv" event={"ID":"2a8466b8-2d52-4b4a-85a7-a566b8ad691d","Type":"ContainerDied","Data":"06b2fc560aafd7dbd5c6fc8be505891036bd9be1ab401be5d7e11e1d8ca130a4"} Nov 02 17:51:33.495527 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.495432 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06b2fc560aafd7dbd5c6fc8be505891036bd9be1ab401be5d7e11e1d8ca130a4" Nov 02 17:51:33.497206 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.497179 4310 generic.go:356] "Generic (PLEG): container finished" podID="fb63e790-c7bc-4312-9afb-30049d3bb405" containerID="a7aa8b5a1125c0ff17e6575affb256865f2ffdadb5caed2fcc1d4c812f499beb" exitCode=0 Nov 02 17:51:33.497346 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.497274 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" event={"ID":"fb63e790-c7bc-4312-9afb-30049d3bb405","Type":"ContainerDied","Data":"a7aa8b5a1125c0ff17e6575affb256865f2ffdadb5caed2fcc1d4c812f499beb"} Nov 02 17:51:33.498950 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.498928 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" Nov 02 17:51:33.499062 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.498952 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn" event={"ID":"266551f9-492e-4113-ac11-5f7fc6ba0304","Type":"ContainerDied","Data":"40450020db90a1e53bbaf4dff9fef87ee131f6248a2cc23a005773cbaac8885c"} Nov 02 17:51:33.499062 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.498992 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40450020db90a1e53bbaf4dff9fef87ee131f6248a2cc23a005773cbaac8885c" Nov 02 17:51:33.517166 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.517113 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podStartSLOduration=1.5170954349999999 podStartE2EDuration="1.517095435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:33.516358789 +0000 UTC m=+690.959441427" watchObservedRunningTime="2026-11-02 17:51:33.517095435 +0000 UTC m=+690.960178070" Nov 02 17:51:33.531959 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:51:33.531924 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:51:33.998129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:33.998068 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:34.395243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.395160 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:34.657861 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.657784 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:34.759247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.759213 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb63e790-c7bc-4312-9afb-30049d3bb405-kube-api-access-6bd66\" (UniqueName: \"kubernetes.io/projected/fb63e790-c7bc-4312-9afb-30049d3bb405-kube-api-access-6bd66\") pod \"fb63e790-c7bc-4312-9afb-30049d3bb405\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " Nov 02 17:51:34.759427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.759254 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-openshift-ca\") pod \"fb63e790-c7bc-4312-9afb-30049d3bb405\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " Nov 02 17:51:34.759554 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.759469 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-xks-ca\") pod \"fb63e790-c7bc-4312-9afb-30049d3bb405\" (UID: \"fb63e790-c7bc-4312-9afb-30049d3bb405\") " Nov 02 17:51:34.759718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.759688 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-openshift-ca" pod "fb63e790-c7bc-4312-9afb-30049d3bb405" (UID: "fb63e790-c7bc-4312-9afb-30049d3bb405"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:34.760058 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.760031 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:34.761524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.761498 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb63e790-c7bc-4312-9afb-30049d3bb405-kube-api-access-6bd66" pod "fb63e790-c7bc-4312-9afb-30049d3bb405" (UID: "fb63e790-c7bc-4312-9afb-30049d3bb405"). InnerVolumeSpecName "kube-api-access-6bd66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:34.761630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.761568 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-xks-ca" pod "fb63e790-c7bc-4312-9afb-30049d3bb405" (UID: "fb63e790-c7bc-4312-9afb-30049d3bb405"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:34.861301 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.861265 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb63e790-c7bc-4312-9afb-30049d3bb405-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:34.861301 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.861293 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bd66\" (UniqueName: \"kubernetes.io/projected/fb63e790-c7bc-4312-9afb-30049d3bb405-kube-api-access-6bd66\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:34.931503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:34.931413 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:35.516906 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:35.516873 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" Nov 02 17:51:35.517128 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:35.516900 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn" event={"ID":"fb63e790-c7bc-4312-9afb-30049d3bb405","Type":"ContainerDied","Data":"eb722760fd01990319616a9c09926e827554c3f35b4c3945d4480983761a01cb"} Nov 02 17:51:35.517128 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:35.516958 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb722760fd01990319616a9c09926e827554c3f35b4c3945d4480983761a01cb" Nov 02 17:51:35.975722 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:35.975585 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:36.990190 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:36.990153 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb"] Nov 02 17:51:36.991583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:36.991555 4310 cpu_manager.go:414] "containerMap: removing container" podUID="2a8466b8-2d52-4b4a-85a7-a566b8ad691d" containerName="revoke-keys" Nov 02 17:51:36.998110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:36.991709 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8466b8-2d52-4b4a-85a7-a566b8ad691d" containerName="revoke-keys" Nov 02 17:51:37.141446 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.141409 4310 cpu_manager.go:414] "containerMap: removing container" podUID="fb63e790-c7bc-4312-9afb-30049d3bb405" containerName="revoke-keys" Nov 02 17:51:37.141760 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.141738 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb63e790-c7bc-4312-9afb-30049d3bb405" containerName="revoke-keys" Nov 02 17:51:37.141935 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.141919 4310 cpu_manager.go:414] "containerMap: removing container" podUID="266551f9-492e-4113-ac11-5f7fc6ba0304" containerName="revoke-keys" Nov 02 17:51:37.142048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.142033 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266551f9-492e-4113-ac11-5f7fc6ba0304" containerName="revoke-keys" Nov 02 17:51:37.142595 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.142576 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb63e790-c7bc-4312-9afb-30049d3bb405" containerName="revoke-keys" Nov 02 17:51:37.142796 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.142781 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="266551f9-492e-4113-ac11-5f7fc6ba0304" containerName="revoke-keys" Nov 02 17:51:37.142918 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.142905 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8466b8-2d52-4b4a-85a7-a566b8ad691d" containerName="revoke-keys" Nov 02 17:51:37.285279 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.285191 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb"] Nov 02 17:51:37.285690 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.285664 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.412102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.388487 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.412102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.388543 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76m8b\" (UniqueName: \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-kube-api-access-76m8b\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.412102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.388590 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.412102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.388668 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.412102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.388707 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.412102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.388738 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f164162-c2b6-41de-9068-ca6eedf8417f-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.412102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.388791 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.412102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.388902 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.412102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.388927 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.503261 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.503223 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.560161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.533180 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.560161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.533308 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.560161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.533343 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76m8b\" (UniqueName: \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-kube-api-access-76m8b\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.560161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.533398 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.560161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.533495 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.560161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.533539 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.560161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.533573 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f164162-c2b6-41de-9068-ca6eedf8417f-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.560161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.533634 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.560161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.503981 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.560161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.534529 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.560161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.534806 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.605155 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.585511 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.605155 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.587599 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f164162-c2b6-41de-9068-ca6eedf8417f-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.658612 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.658500 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.872453 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.846253 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.873282 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.873249 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:37.878104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:37.874118 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76m8b\" (UniqueName: \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-kube-api-access-76m8b\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:38.021327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.021292 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:38.426414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.426345 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb"] Nov 02 17:51:38.426815 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:38.426791 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f164162_c2b6_41de_9068_ca6eedf8417f.slice/crio-55714b404783ae5bd22a46b8172571ca541c2bd78a26dc9d3f02d710978064f5 WatchSource:0}: Error finding container 55714b404783ae5bd22a46b8172571ca541c2bd78a26dc9d3f02d710978064f5: Status 404 returned error can't find the container with id 55714b404783ae5bd22a46b8172571ca541c2bd78a26dc9d3f02d710978064f5 Nov 02 17:51:38.429456 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.429425 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:38.429536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.429488 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:38.429536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.429520 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:38.520277 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.520238 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:38.728370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.728335 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" event={"ID":"3f164162-c2b6-41de-9068-ca6eedf8417f","Type":"ContainerStarted","Data":"ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e"} Nov 02 17:51:38.728568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.728389 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" event={"ID":"3f164162-c2b6-41de-9068-ca6eedf8417f","Type":"ContainerStarted","Data":"55714b404783ae5bd22a46b8172571ca541c2bd78a26dc9d3f02d710978064f5"} Nov 02 17:51:38.785876 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.785806 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podStartSLOduration=2.785766047 podStartE2EDuration="2.785766047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:38.785689432 +0000 UTC m=+696.228772081" watchObservedRunningTime="2026-11-02 17:51:38.785766047 +0000 UTC m=+696.228848684" Nov 02 17:51:38.819211 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.819170 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24"] Nov 02 17:51:38.824704 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.824674 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:38.826640 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.826611 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24"] Nov 02 17:51:38.834780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.834755 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:51:38.835071 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.835013 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:51:38.928708 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.928672 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd412da1-fmj24\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:38.928906 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.928762 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4km74\" (UniqueName: \"kubernetes.io/projected/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-kube-api-access-4km74\") pod \"maas-api-revoke-sub-dd412da1-fmj24\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:38.928906 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:38.928786 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd412da1-fmj24\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:39.022413 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.022246 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:39.024145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.023936 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:51:39.024145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.023979 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:51:39.029409 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.029382 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd412da1-fmj24\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:39.029536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.029510 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4km74\" (UniqueName: \"kubernetes.io/projected/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-kube-api-access-4km74\") pod \"maas-api-revoke-sub-dd412da1-fmj24\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:39.029536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.029532 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd412da1-fmj24\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:39.030450 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.030422 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd412da1-fmj24\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:39.032041 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.032021 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd412da1-fmj24\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:39.042176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.042130 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4km74\" (UniqueName: \"kubernetes.io/projected/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-kube-api-access-4km74\") pod \"maas-api-revoke-sub-dd412da1-fmj24\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:39.216103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.214918 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:39.739098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.739057 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24"] Nov 02 17:51:39.741522 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:39.741488 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd618ff6b_44d2_41cf_af53_86e4f7bf4b3c.slice/crio-08e56f6d9c36ad27180d9a070a7ae2229c03d13c1827cb6e06e7bff1225a9bd1 WatchSource:0}: Error finding container 08e56f6d9c36ad27180d9a070a7ae2229c03d13c1827cb6e06e7bff1225a9bd1: Status 404 returned error can't find the container with id 08e56f6d9c36ad27180d9a070a7ae2229c03d13c1827cb6e06e7bff1225a9bd1 Nov 02 17:51:39.813165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.786581 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:39.836666 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.833569 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc"] Nov 02 17:51:39.895110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.891731 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc"] Nov 02 17:51:39.895110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:39.891874 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:40.023297 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.023262 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:51:40.023691 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.023316 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:51:40.030915 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.030895 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e00247bb-5c6vc\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:40.031051 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.031036 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9929\" (UniqueName: \"kubernetes.io/projected/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-kube-api-access-p9929\") pod \"maas-api-revoke-sub-e00247bb-5c6vc\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:40.031105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.031095 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e00247bb-5c6vc\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:40.132276 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.132240 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9929\" (UniqueName: \"kubernetes.io/projected/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-kube-api-access-p9929\") pod \"maas-api-revoke-sub-e00247bb-5c6vc\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:40.132461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.132309 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e00247bb-5c6vc\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:40.132461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.132356 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e00247bb-5c6vc\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:40.133099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.133064 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e00247bb-5c6vc\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:40.133400 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.133380 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx"] Nov 02 17:51:40.134672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.134651 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e00247bb-5c6vc\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:40.137596 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.137582 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:40.179879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.179837 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9929\" (UniqueName: \"kubernetes.io/projected/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-kube-api-access-p9929\") pod \"maas-api-revoke-sub-e00247bb-5c6vc\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:40.182574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.182119 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx"] Nov 02 17:51:40.233429 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.233397 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9690cd4d-xhxhx\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:40.233558 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.233550 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4hkt\" (UniqueName: \"kubernetes.io/projected/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-kube-api-access-v4hkt\") pod \"maas-api-revoke-sub-9690cd4d-xhxhx\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:40.233594 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.233568 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9690cd4d-xhxhx\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:40.263569 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.261378 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:40.337945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.336071 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4hkt\" (UniqueName: \"kubernetes.io/projected/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-kube-api-access-v4hkt\") pod \"maas-api-revoke-sub-9690cd4d-xhxhx\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:40.337945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.336137 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9690cd4d-xhxhx\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:40.337945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.336204 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9690cd4d-xhxhx\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:40.337945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.337044 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9690cd4d-xhxhx\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:40.341319 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.338569 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9690cd4d-xhxhx\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:40.377108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.376933 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4hkt\" (UniqueName: \"kubernetes.io/projected/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-kube-api-access-v4hkt\") pod \"maas-api-revoke-sub-9690cd4d-xhxhx\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:40.497801 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.497767 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:40.583428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.583338 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc"] Nov 02 17:51:40.589132 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:40.585494 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccdbb63_26f1_4b82_82d2_cf4edb0a499a.slice/crio-8cfa4ef1113859114669d14aa40f380d82935ab5d0a5d90e4eea78b42f012972 WatchSource:0}: Error finding container 8cfa4ef1113859114669d14aa40f380d82935ab5d0a5d90e4eea78b42f012972: Status 404 returned error can't find the container with id 8cfa4ef1113859114669d14aa40f380d82935ab5d0a5d90e4eea78b42f012972 Nov 02 17:51:40.728382 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.728344 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" event={"ID":"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c","Type":"ContainerStarted","Data":"d63de7d3f35389a46fcf2a4684ac1cdd631cc7e46fc91e2333005232a5d0b252"} Nov 02 17:51:40.728575 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.728406 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" event={"ID":"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c","Type":"ContainerStarted","Data":"08e56f6d9c36ad27180d9a070a7ae2229c03d13c1827cb6e06e7bff1225a9bd1"} Nov 02 17:51:40.729696 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.729670 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" event={"ID":"cccdbb63-26f1-4b82-82d2-cf4edb0a499a","Type":"ContainerStarted","Data":"8cfa4ef1113859114669d14aa40f380d82935ab5d0a5d90e4eea78b42f012972"} Nov 02 17:51:40.810774 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.810741 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:40.824770 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.824706 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" podStartSLOduration=2.8246853720000002 podStartE2EDuration="2.824685372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:40.767133267 +0000 UTC m=+698.210215904" watchObservedRunningTime="2026-11-02 17:51:40.824685372 +0000 UTC m=+698.267768009" Nov 02 17:51:40.828010 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:40.827983 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx"] Nov 02 17:51:40.828151 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:40.828001 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a0d902_ffde_4bfb_a5d5_f00fdbdea45a.slice/crio-517cc354762b53646a6a28d93929622a0a8dffe4771bf838c95a8e6fb151d93d WatchSource:0}: Error finding container 517cc354762b53646a6a28d93929622a0a8dffe4771bf838c95a8e6fb151d93d: Status 404 returned error can't find the container with id 517cc354762b53646a6a28d93929622a0a8dffe4771bf838c95a8e6fb151d93d Nov 02 17:51:41.023149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:41.023116 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:51:41.023295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:41.023171 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:51:41.737292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:41.737262 4310 generic.go:356] "Generic (PLEG): container finished" podID="cccdbb63-26f1-4b82-82d2-cf4edb0a499a" containerID="bf74db187461117a124b9e0af96e12f879ca3f736d3f439d0e8a85b0a248bfce" exitCode=0 Nov 02 17:51:41.737774 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:41.737352 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" event={"ID":"cccdbb63-26f1-4b82-82d2-cf4edb0a499a","Type":"ContainerDied","Data":"bf74db187461117a124b9e0af96e12f879ca3f736d3f439d0e8a85b0a248bfce"} Nov 02 17:51:41.739123 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:41.739099 4310 generic.go:356] "Generic (PLEG): container finished" podID="d618ff6b-44d2-41cf-af53-86e4f7bf4b3c" containerID="d63de7d3f35389a46fcf2a4684ac1cdd631cc7e46fc91e2333005232a5d0b252" exitCode=0 Nov 02 17:51:41.739271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:41.739184 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" event={"ID":"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c","Type":"ContainerDied","Data":"d63de7d3f35389a46fcf2a4684ac1cdd631cc7e46fc91e2333005232a5d0b252"} Nov 02 17:51:41.740980 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:41.740952 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" event={"ID":"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a","Type":"ContainerStarted","Data":"da02560a361b8e5c0e4d4959f7979d56da676bd666ea6bc98b0ac9cb3f384b68"} Nov 02 17:51:41.741108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:41.741001 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" event={"ID":"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a","Type":"ContainerStarted","Data":"517cc354762b53646a6a28d93929622a0a8dffe4771bf838c95a8e6fb151d93d"} Nov 02 17:51:41.792797 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:41.792743 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" podStartSLOduration=1.792728959 podStartE2EDuration="1.792728959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:41.790123072 +0000 UTC m=+699.233205712" watchObservedRunningTime="2026-11-02 17:51:41.792728959 +0000 UTC m=+699.235811595" Nov 02 17:51:42.022491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.022456 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:51:42.022636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.022508 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:51:42.034986 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.034955 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:42.395752 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.395715 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:42.410321 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.410289 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:42.420310 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.420276 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9"] Nov 02 17:51:42.424469 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.424451 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:42.436369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.436335 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9"] Nov 02 17:51:42.492029 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.491987 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720e2077-v8td9\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:42.492263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.492071 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720e2077-v8td9\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:42.492263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.492144 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tq5\" (UniqueName: \"kubernetes.io/projected/a3439928-8ca9-4d71-8fe2-8bfb959520ef-kube-api-access-q2tq5\") pod \"maas-api-revoke-sub-720e2077-v8td9\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:42.593711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.593674 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720e2077-v8td9\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:42.593883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.593719 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720e2077-v8td9\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:42.593883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.593757 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tq5\" (UniqueName: \"kubernetes.io/projected/a3439928-8ca9-4d71-8fe2-8bfb959520ef-kube-api-access-q2tq5\") pod \"maas-api-revoke-sub-720e2077-v8td9\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:42.594348 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.594316 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720e2077-v8td9\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:42.596240 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.596208 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720e2077-v8td9\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:42.603170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.603143 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tq5\" (UniqueName: \"kubernetes.io/projected/a3439928-8ca9-4d71-8fe2-8bfb959520ef-kube-api-access-q2tq5\") pod \"maas-api-revoke-sub-720e2077-v8td9\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:42.741003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.740974 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:42.777116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.777073 4310 generic.go:356] "Generic (PLEG): container finished" podID="d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a" containerID="da02560a361b8e5c0e4d4959f7979d56da676bd666ea6bc98b0ac9cb3f384b68" exitCode=0 Nov 02 17:51:42.777233 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.777171 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" event={"ID":"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a","Type":"ContainerDied","Data":"da02560a361b8e5c0e4d4959f7979d56da676bd666ea6bc98b0ac9cb3f384b68"} Nov 02 17:51:42.792348 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:42.792321 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:51:43.023153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.023056 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:51:43.023153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.023136 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:51:43.081361 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.078804 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9"] Nov 02 17:51:43.083111 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:43.082392 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3439928_8ca9_4d71_8fe2_8bfb959520ef.slice/crio-67909aa22686801f4a6a74e32bf62ca65ac20df3808be3ef436e20bb5674be2e WatchSource:0}: Error finding container 67909aa22686801f4a6a74e32bf62ca65ac20df3808be3ef436e20bb5674be2e: Status 404 returned error can't find the container with id 67909aa22686801f4a6a74e32bf62ca65ac20df3808be3ef436e20bb5674be2e Nov 02 17:51:43.111595 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.111572 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:43.200849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.200680 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-kube-api-access-4km74\" (UniqueName: \"kubernetes.io/projected/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-kube-api-access-4km74\") pod \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " Nov 02 17:51:43.200849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.200833 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-xks-ca\") pod \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " Nov 02 17:51:43.201123 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.200902 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-openshift-ca\") pod \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\" (UID: \"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c\") " Nov 02 17:51:43.201567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.201539 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-openshift-ca" pod "d618ff6b-44d2-41cf-af53-86e4f7bf4b3c" (UID: "d618ff6b-44d2-41cf-af53-86e4f7bf4b3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:43.205509 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.205461 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:43.261735 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.261365 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-kube-api-access-4km74" pod "d618ff6b-44d2-41cf-af53-86e4f7bf4b3c" (UID: "d618ff6b-44d2-41cf-af53-86e4f7bf4b3c"). InnerVolumeSpecName "kube-api-access-4km74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:43.279277 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.276764 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-xks-ca" pod "d618ff6b-44d2-41cf-af53-86e4f7bf4b3c" (UID: "d618ff6b-44d2-41cf-af53-86e4f7bf4b3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:43.313657 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.313465 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4km74\" (UniqueName: \"kubernetes.io/projected/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-kube-api-access-4km74\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:43.313916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.313894 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:43.314048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.314028 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:43.316553 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.316526 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:43.414963 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.414929 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-kube-api-access-p9929\" (UniqueName: \"kubernetes.io/projected/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-kube-api-access-p9929\") pod \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " Nov 02 17:51:43.415173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.415014 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-openshift-ca\") pod \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " Nov 02 17:51:43.415300 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.415279 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-xks-ca\") pod \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\" (UID: \"cccdbb63-26f1-4b82-82d2-cf4edb0a499a\") " Nov 02 17:51:43.415436 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.415413 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-openshift-ca" pod "cccdbb63-26f1-4b82-82d2-cf4edb0a499a" (UID: "cccdbb63-26f1-4b82-82d2-cf4edb0a499a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:43.415708 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.415692 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:43.417226 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.417202 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-xks-ca" pod "cccdbb63-26f1-4b82-82d2-cf4edb0a499a" (UID: "cccdbb63-26f1-4b82-82d2-cf4edb0a499a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:43.417314 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.417293 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-kube-api-access-p9929" pod "cccdbb63-26f1-4b82-82d2-cf4edb0a499a" (UID: "cccdbb63-26f1-4b82-82d2-cf4edb0a499a"). InnerVolumeSpecName "kube-api-access-p9929". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:43.517209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.517175 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9929\" (UniqueName: \"kubernetes.io/projected/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-kube-api-access-p9929\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:43.517209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.517209 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccdbb63-26f1-4b82-82d2-cf4edb0a499a-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:43.785913 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.785878 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" Nov 02 17:51:43.786373 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.785913 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24" event={"ID":"d618ff6b-44d2-41cf-af53-86e4f7bf4b3c","Type":"ContainerDied","Data":"08e56f6d9c36ad27180d9a070a7ae2229c03d13c1827cb6e06e7bff1225a9bd1"} Nov 02 17:51:43.786373 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.785963 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08e56f6d9c36ad27180d9a070a7ae2229c03d13c1827cb6e06e7bff1225a9bd1" Nov 02 17:51:43.787467 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.787446 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" Nov 02 17:51:43.787605 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.787448 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc" event={"ID":"cccdbb63-26f1-4b82-82d2-cf4edb0a499a","Type":"ContainerDied","Data":"8cfa4ef1113859114669d14aa40f380d82935ab5d0a5d90e4eea78b42f012972"} Nov 02 17:51:43.787605 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.787581 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cfa4ef1113859114669d14aa40f380d82935ab5d0a5d90e4eea78b42f012972" Nov 02 17:51:43.789246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.789219 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" event={"ID":"a3439928-8ca9-4d71-8fe2-8bfb959520ef","Type":"ContainerStarted","Data":"c66cc3b04c55c138792c16d3a34795e7359e2b09b9ca5ef3f91baf799cef01e9"} Nov 02 17:51:43.789365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.789265 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" event={"ID":"a3439928-8ca9-4d71-8fe2-8bfb959520ef","Type":"ContainerStarted","Data":"67909aa22686801f4a6a74e32bf62ca65ac20df3808be3ef436e20bb5674be2e"} Nov 02 17:51:43.817028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.816977 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" podStartSLOduration=1.816961842 podStartE2EDuration="1.816961842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:43.814621197 +0000 UTC m=+701.257703835" watchObservedRunningTime="2026-11-02 17:51:43.816961842 +0000 UTC m=+701.260044478" Nov 02 17:51:43.967103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:43.957305 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:44.023056 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.023021 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:51:44.023056 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.023053 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-xks-ca\") pod \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " Nov 02 17:51:44.023385 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.023067 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:51:44.023385 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.023185 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-kube-api-access-v4hkt\" (UniqueName: \"kubernetes.io/projected/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-kube-api-access-v4hkt\") pod \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " Nov 02 17:51:44.023385 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.023237 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-openshift-ca\") pod \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\" (UID: \"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a\") " Nov 02 17:51:44.023606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.023584 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-openshift-ca" pod "d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a" (UID: "d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:44.023819 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.023794 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:44.025358 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.025334 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-xks-ca" pod "d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a" (UID: "d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:44.025472 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.025423 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-kube-api-access-v4hkt" pod "d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a" (UID: "d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a"). InnerVolumeSpecName "kube-api-access-v4hkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:44.125487 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.125394 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4hkt\" (UniqueName: \"kubernetes.io/projected/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-kube-api-access-v4hkt\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:44.125487 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.125430 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:44.164104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.159568 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:44.798846 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.798815 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" Nov 02 17:51:44.799268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.798843 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx" event={"ID":"d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a","Type":"ContainerDied","Data":"517cc354762b53646a6a28d93929622a0a8dffe4771bf838c95a8e6fb151d93d"} Nov 02 17:51:44.799268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.798893 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="517cc354762b53646a6a28d93929622a0a8dffe4771bf838c95a8e6fb151d93d" Nov 02 17:51:44.800860 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.800835 4310 generic.go:356] "Generic (PLEG): container finished" podID="a3439928-8ca9-4d71-8fe2-8bfb959520ef" containerID="c66cc3b04c55c138792c16d3a34795e7359e2b09b9ca5ef3f91baf799cef01e9" exitCode=0 Nov 02 17:51:44.801017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:44.800979 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" event={"ID":"a3439928-8ca9-4d71-8fe2-8bfb959520ef","Type":"ContainerDied","Data":"c66cc3b04c55c138792c16d3a34795e7359e2b09b9ca5ef3f91baf799cef01e9"} Nov 02 17:51:45.023385 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.023142 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 17:51:45.023385 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.023216 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 17:51:45.056294 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.056257 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4"] Nov 02 17:51:45.056827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.056811 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a" containerName="revoke-keys" Nov 02 17:51:45.056827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.056825 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a" containerName="revoke-keys" Nov 02 17:51:45.056973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.056855 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d618ff6b-44d2-41cf-af53-86e4f7bf4b3c" containerName="revoke-keys" Nov 02 17:51:45.056973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.056860 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d618ff6b-44d2-41cf-af53-86e4f7bf4b3c" containerName="revoke-keys" Nov 02 17:51:45.056973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.056876 4310 cpu_manager.go:414] "containerMap: removing container" podUID="cccdbb63-26f1-4b82-82d2-cf4edb0a499a" containerName="revoke-keys" Nov 02 17:51:45.056973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.056881 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cccdbb63-26f1-4b82-82d2-cf4edb0a499a" containerName="revoke-keys" Nov 02 17:51:45.056973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.056949 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="cccdbb63-26f1-4b82-82d2-cf4edb0a499a" containerName="revoke-keys" Nov 02 17:51:45.056973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.056959 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a" containerName="revoke-keys" Nov 02 17:51:45.056973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.056970 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d618ff6b-44d2-41cf-af53-86e4f7bf4b3c" containerName="revoke-keys" Nov 02 17:51:45.060800 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.060774 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.066167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.066143 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ea3867\"/\"default-dockercfg-mb25d\"" Nov 02 17:51:45.066167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.066143 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ea3867\"/\"distinct-w5-ea3867-kserve-self-signed-certs\"" Nov 02 17:51:45.067308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.067285 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ea3867\"/\"openshift-service-ca.crt\"" Nov 02 17:51:45.067429 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.067342 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ea3867\"/\"kube-root-ca.crt\"" Nov 02 17:51:45.078948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.078915 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4"] Nov 02 17:51:45.111515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.111474 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:45.137003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.136960 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-home\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.137227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.137013 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tmp-dir\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.137227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.137193 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tls-certs\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.137227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.137213 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tvdt\" (UniqueName: \"kubernetes.io/projected/80e9e16c-4460-4acf-8c44-bae8d73e09dd-kube-api-access-8tvdt\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.137387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.137247 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-dshm\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.137387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.137266 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-model-cache\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.239001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.238949 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-home\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.252262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.252216 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tmp-dir\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.252636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.252615 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tls-certs\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.252798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.252775 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tvdt\" (UniqueName: \"kubernetes.io/projected/80e9e16c-4460-4acf-8c44-bae8d73e09dd-kube-api-access-8tvdt\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.252959 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.252941 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-dshm\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.253097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.253065 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-model-cache\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.253682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.253660 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-model-cache\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.281222 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.242479 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-home\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.281467 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.257441 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tmp-dir\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.295515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.293175 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tvdt\" (UniqueName: \"kubernetes.io/projected/80e9e16c-4460-4acf-8c44-bae8d73e09dd-kube-api-access-8tvdt\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.295515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.293898 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:45.296906 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.296873 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tls-certs\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.297589 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.297563 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-dshm\") pod \"distinct-w5-ea3867-kserve-65bf875d78-fwxs4\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.374730 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.374694 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:45.711966 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:45.711917 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80e9e16c_4460_4acf_8c44_bae8d73e09dd.slice/crio-e8bf3965e3f78b529b353e76e4ecf8841b9ce84657d3daa18d4b6a8084e51f90 WatchSource:0}: Error finding container e8bf3965e3f78b529b353e76e4ecf8841b9ce84657d3daa18d4b6a8084e51f90: Status 404 returned error can't find the container with id e8bf3965e3f78b529b353e76e4ecf8841b9ce84657d3daa18d4b6a8084e51f90 Nov 02 17:51:45.712655 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.712625 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4"] Nov 02 17:51:45.810250 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.810218 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" event={"ID":"80e9e16c-4460-4acf-8c44-bae8d73e09dd","Type":"ContainerStarted","Data":"2f137990694d17a26e14c43eab84468233d19306643fcb84a0ea46ac3d6a0ecd"} Nov 02 17:51:45.810804 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.810259 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" event={"ID":"80e9e16c-4460-4acf-8c44-bae8d73e09dd","Type":"ContainerStarted","Data":"e8bf3965e3f78b529b353e76e4ecf8841b9ce84657d3daa18d4b6a8084e51f90"} Nov 02 17:51:45.825430 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.825380 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:45.835975 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.835913 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podStartSLOduration=0.835890793 podStartE2EDuration="835.890793ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:45.833406785 +0000 UTC m=+703.276489421" watchObservedRunningTime="2026-11-02 17:51:45.835890793 +0000 UTC m=+703.278973429" Nov 02 17:51:45.947890 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.947850 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:45.977394 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.977353 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-xks-ca\") pod \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " Nov 02 17:51:45.977585 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.977425 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-openshift-ca\") pod \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " Nov 02 17:51:45.977585 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.977461 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3439928-8ca9-4d71-8fe2-8bfb959520ef-kube-api-access-q2tq5\" (UniqueName: \"kubernetes.io/projected/a3439928-8ca9-4d71-8fe2-8bfb959520ef-kube-api-access-q2tq5\") pod \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\" (UID: \"a3439928-8ca9-4d71-8fe2-8bfb959520ef\") " Nov 02 17:51:45.977833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.977807 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-openshift-ca" pod "a3439928-8ca9-4d71-8fe2-8bfb959520ef" (UID: "a3439928-8ca9-4d71-8fe2-8bfb959520ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:45.977920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.977899 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:45.979850 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.979820 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-xks-ca" pod "a3439928-8ca9-4d71-8fe2-8bfb959520ef" (UID: "a3439928-8ca9-4d71-8fe2-8bfb959520ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:45.979959 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:45.979850 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3439928-8ca9-4d71-8fe2-8bfb959520ef-kube-api-access-q2tq5" pod "a3439928-8ca9-4d71-8fe2-8bfb959520ef" (UID: "a3439928-8ca9-4d71-8fe2-8bfb959520ef"). InnerVolumeSpecName "kube-api-access-q2tq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:46.034118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.029208 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:46.034118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.030845 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:46.034118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.031866 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 17:51:46.108437 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.107667 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3439928-8ca9-4d71-8fe2-8bfb959520ef-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:46.108437 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.107701 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2tq5\" (UniqueName: \"kubernetes.io/projected/a3439928-8ca9-4d71-8fe2-8bfb959520ef-kube-api-access-q2tq5\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:46.391101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.387207 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:46.421826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.421786 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:46.785105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.763702 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5"] Nov 02 17:51:46.786560 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.786523 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a3439928-8ca9-4d71-8fe2-8bfb959520ef" containerName="revoke-keys" Nov 02 17:51:46.786739 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.786722 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3439928-8ca9-4d71-8fe2-8bfb959520ef" containerName="revoke-keys" Nov 02 17:51:46.787125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.787105 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3439928-8ca9-4d71-8fe2-8bfb959520ef" containerName="revoke-keys" Nov 02 17:51:46.896102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.870454 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5"] Nov 02 17:51:46.916239 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.886278 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:46.940143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.926779 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:46.940143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.926860 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:46.940143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.926956 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:46.940143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.927009 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fbt7\" (UniqueName: \"kubernetes.io/projected/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-kube-api-access-4fbt7\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:46.940143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.927059 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:46.940143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.927155 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:46.940143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.927205 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:46.940143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.927244 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:46.940143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.927273 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:46.972559 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.972529 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" Nov 02 17:51:46.996327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.996291 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9" event={"ID":"a3439928-8ca9-4d71-8fe2-8bfb959520ef","Type":"ContainerDied","Data":"67909aa22686801f4a6a74e32bf62ca65ac20df3808be3ef436e20bb5674be2e"} Nov 02 17:51:46.996571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:46.996553 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67909aa22686801f4a6a74e32bf62ca65ac20df3808be3ef436e20bb5674be2e" Nov 02 17:51:47.332116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.326706 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.332116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.326778 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.332116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.326818 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.332116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.326900 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.332116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.326992 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.338351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.338319 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.425104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.421739 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.425404 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.425369 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fbt7\" (UniqueName: \"kubernetes.io/projected/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-kube-api-access-4fbt7\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.425634 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.425613 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.425878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.425856 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.431099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.427067 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.443105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.433219 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.445132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.445105 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.447107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.446345 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.513774 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.513731 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.625159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.623552 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.625159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.624021 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.668672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.653615 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fbt7\" (UniqueName: \"kubernetes.io/projected/d1be36c1-1e0e-478d-bdeb-7a9704cee2e0-kube-api-access-4fbt7\") pod \"maas-default-gateway-openshift-default-fcb75c45f-l98w5\" (UID: \"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:47.844634 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.844577 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:51:47.917738 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:47.917628 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:48.304261 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:48.304227 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:48.418983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:48.418942 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5"] Nov 02 17:51:48.419712 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:51:48.419648 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1be36c1_1e0e_478d_bdeb_7a9704cee2e0.slice/crio-d1998a35ba30f0df6c0fda4524fdcb5d82f96005df4834c0158ee3dc1ac5884e WatchSource:0}: Error finding container d1998a35ba30f0df6c0fda4524fdcb5d82f96005df4834c0158ee3dc1ac5884e: Status 404 returned error can't find the container with id d1998a35ba30f0df6c0fda4524fdcb5d82f96005df4834c0158ee3dc1ac5884e Nov 02 17:51:48.422217 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:48.422187 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:48.422284 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:48.422249 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:48.422284 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:48.422277 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:48.998499 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:48.998466 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" event={"ID":"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0","Type":"ContainerStarted","Data":"8fb3b6c1ee84f56db0b9355f51563bedce4d3656bc8392d2921017ed851d90af"} Nov 02 17:51:48.998895 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:48.998520 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" event={"ID":"d1be36c1-1e0e-478d-bdeb-7a9704cee2e0","Type":"ContainerStarted","Data":"d1998a35ba30f0df6c0fda4524fdcb5d82f96005df4834c0158ee3dc1ac5884e"} Nov 02 17:51:49.027416 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:49.027370 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podStartSLOduration=3.027355536 podStartE2EDuration="3.027355536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:49.024759421 +0000 UTC m=+706.467842048" watchObservedRunningTime="2026-11-02 17:51:49.027355536 +0000 UTC m=+706.470438171" Nov 02 17:51:49.170625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:49.170567 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:49.918855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:49.918821 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:49.920086 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:49.920054 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 17:51:49.920219 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:49.920121 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 17:51:50.171664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:50.171549 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:50.468523 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:50.468495 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:50.919193 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:50.919110 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 17:51:50.919193 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:50.919164 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 17:51:51.367104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:51.360491 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:51.919877 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:51.919832 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 17:51:51.920032 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:51.919895 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 17:51:52.403761 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.403725 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2"] Nov 02 17:51:52.408324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.408303 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.433658 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.433631 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.433814 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.433669 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.433814 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.433781 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.433939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.433825 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0253d943-e554-4580-b042-cabae8436768-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.433939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.433880 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.433939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.433906 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0253d943-e554-4580-b042-cabae8436768-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.434038 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.433994 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnk78\" (UniqueName: \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-kube-api-access-jnk78\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.434118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.434104 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.434182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.434169 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.496847 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.496812 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2"] Nov 02 17:51:52.535037 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.534984 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.535247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.535096 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.535247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.535169 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.535247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.535199 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.535443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.535257 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.535443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.535284 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0253d943-e554-4580-b042-cabae8436768-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.535443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.535311 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.535443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.535328 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0253d943-e554-4580-b042-cabae8436768-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.535443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.535349 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnk78\" (UniqueName: \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-kube-api-access-jnk78\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.535654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.535536 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.535944 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.535914 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.536057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.535950 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.536351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.536324 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0253d943-e554-4580-b042-cabae8436768-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.536481 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.536457 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.538198 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.538175 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.538630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.538601 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0253d943-e554-4580-b042-cabae8436768-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.576174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.576143 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnk78\" (UniqueName: \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-kube-api-access-jnk78\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.576396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.576284 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.671235 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.671131 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc"] Nov 02 17:51:52.681632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.681583 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.701592 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.701483 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc"] Nov 02 17:51:52.724317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.724274 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:52.737653 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.737625 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.737839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.737767 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.737909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.737844 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.737909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.737900 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.738027 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.738009 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.738133 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.738050 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1801c5bf-b709-48b0-9e6b-721f85eb4520-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.738133 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.738117 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.738252 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.738153 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.738252 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.738183 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfkmj\" (UniqueName: \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-kube-api-access-wfkmj\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.839777 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.839743 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.839915 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.839784 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1801c5bf-b709-48b0-9e6b-721f85eb4520-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.839915 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.839805 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.839915 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.839824 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.839915 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.839841 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfkmj\" (UniqueName: \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-kube-api-access-wfkmj\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.839915 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.839879 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.840090 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.839922 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.840090 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.839962 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.840090 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.839993 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.840371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.840355 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.841387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.841045 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.841387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.841255 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.841387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.841330 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.841885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.841862 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1801c5bf-b709-48b0-9e6b-721f85eb4520-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.865131 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.856889 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.869337 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.869275 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.876201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.876167 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfkmj\" (UniqueName: \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-kube-api-access-wfkmj\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.876763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.876663 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:52.919506 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.919475 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 17:51:52.919722 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:52.919700 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 17:51:53.018012 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.017979 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:53.095382 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.092544 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:53.095382 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.093939 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:53.095382 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.094021 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:53.095382 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.094063 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:53.101131 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.101013 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2"] Nov 02 17:51:53.387273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.387197 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b"] Nov 02 17:51:53.392352 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.392294 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:53.399894 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.399344 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:51:53.399894 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.399516 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:51:53.400952 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.400414 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b"] Nov 02 17:51:53.426799 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.426616 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:53.426799 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.426701 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:53.426799 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.426742 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:51:53.427738 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.427677 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc"] Nov 02 17:51:53.477127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.455993 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ef14ca-drt2b\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:53.477127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.456156 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdm2\" (UniqueName: \"kubernetes.io/projected/81e92a80-24f9-4a24-b6b2-5e53b21879b2-kube-api-access-khdm2\") pod \"maas-api-revoke-sub-c6ef14ca-drt2b\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:53.477127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.456351 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ef14ca-drt2b\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:53.586103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.571977 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdm2\" (UniqueName: \"kubernetes.io/projected/81e92a80-24f9-4a24-b6b2-5e53b21879b2-kube-api-access-khdm2\") pod \"maas-api-revoke-sub-c6ef14ca-drt2b\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:53.586103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.572147 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ef14ca-drt2b\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:53.586103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.572218 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ef14ca-drt2b\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:53.586103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.573604 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ef14ca-drt2b\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:53.586929 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.586874 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ef14ca-drt2b\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:53.596611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.596585 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdm2\" (UniqueName: \"kubernetes.io/projected/81e92a80-24f9-4a24-b6b2-5e53b21879b2-kube-api-access-khdm2\") pod \"maas-api-revoke-sub-c6ef14ca-drt2b\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:53.715089 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.715054 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:53.919701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.919659 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 17:51:53.919891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:53.919723 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 17:51:54.033965 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.033877 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" event={"ID":"1801c5bf-b709-48b0-9e6b-721f85eb4520","Type":"ContainerStarted","Data":"0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5"} Nov 02 17:51:54.034127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.033972 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" event={"ID":"1801c5bf-b709-48b0-9e6b-721f85eb4520","Type":"ContainerStarted","Data":"ed6923ab65f948a3bbeeb95c3713cb2f7437a20b9c50b15733f88210cb9c6cba"} Nov 02 17:51:54.035660 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.035631 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" event={"ID":"0253d943-e554-4580-b042-cabae8436768","Type":"ContainerStarted","Data":"5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6"} Nov 02 17:51:54.035774 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.035677 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" event={"ID":"0253d943-e554-4580-b042-cabae8436768","Type":"ContainerStarted","Data":"579c053d784bcc61bed9c4dc0c46a350659676babef4a19bb190dc05783ffeb5"} Nov 02 17:51:54.077287 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.077200 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b"] Nov 02 17:51:54.079778 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.079719 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podStartSLOduration=2.079698566 podStartE2EDuration="2.079698566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:54.070442406 +0000 UTC m=+711.513525066" watchObservedRunningTime="2026-11-02 17:51:54.079698566 +0000 UTC m=+711.522781203" Nov 02 17:51:54.108659 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.108605 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podStartSLOduration=2.108585691 podStartE2EDuration="2.108585691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:54.100975857 +0000 UTC m=+711.544058515" watchObservedRunningTime="2026-11-02 17:51:54.108585691 +0000 UTC m=+711.551668330" Nov 02 17:51:54.193103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.179736 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:54.725537 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.725504 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:51:54.726891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.726869 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:51:54.727002 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.726920 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:51:54.919116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.919064 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 17:51:54.919268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:54.919133 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 17:51:55.018857 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.018777 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:51:55.020678 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.020651 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:51:55.020813 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.020701 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 17:51:55.071436 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.066805 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" event={"ID":"81e92a80-24f9-4a24-b6b2-5e53b21879b2","Type":"ContainerStarted","Data":"523c570822321d2e6da5c30d91cb5171b809ab43fdbec6f3f27c7b89a5156ac2"} Nov 02 17:51:55.071436 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.066888 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" event={"ID":"81e92a80-24f9-4a24-b6b2-5e53b21879b2","Type":"ContainerStarted","Data":"f46522694000ee1042755e319356233d8491d20636bb61baeac50683fc5bc1fa"} Nov 02 17:51:55.376104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.375834 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:55.413909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.413872 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:55.439735 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.439679 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" podStartSLOduration=2.439664339 podStartE2EDuration="2.439664339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:55.104244883 +0000 UTC m=+712.547327547" watchObservedRunningTime="2026-11-02 17:51:55.439664339 +0000 UTC m=+712.882747303" Nov 02 17:51:55.465102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.464614 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:55.724779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.724730 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:51:55.724971 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.724812 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:51:55.919527 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.919491 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 17:51:55.919948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:55.919545 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 17:51:56.018410 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:56.018319 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:51:56.018635 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:56.018601 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 17:51:56.104555 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:56.104507 4310 generic.go:356] "Generic (PLEG): container finished" podID="81e92a80-24f9-4a24-b6b2-5e53b21879b2" containerID="523c570822321d2e6da5c30d91cb5171b809ab43fdbec6f3f27c7b89a5156ac2" exitCode=0 Nov 02 17:51:56.105191 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:56.105160 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" event={"ID":"81e92a80-24f9-4a24-b6b2-5e53b21879b2","Type":"ContainerDied","Data":"523c570822321d2e6da5c30d91cb5171b809ab43fdbec6f3f27c7b89a5156ac2"} Nov 02 17:51:56.122536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:56.122508 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:51:56.543102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:56.539174 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:56.724905 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:56.724873 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:51:56.725227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:56.725199 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:51:57.020770 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.020268 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:51:57.020770 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.020329 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 17:51:57.424574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.424454 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:57.628203 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.626898 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:57.658692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.658660 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:57.750826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.750782 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:51:57.751042 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.750919 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:51:57.756103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.752470 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-openshift-ca\") pod \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " Nov 02 17:51:57.774307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.774270 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-xks-ca\") pod \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " Nov 02 17:51:57.774605 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.774587 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e92a80-24f9-4a24-b6b2-5e53b21879b2-kube-api-access-khdm2\" (UniqueName: \"kubernetes.io/projected/81e92a80-24f9-4a24-b6b2-5e53b21879b2-kube-api-access-khdm2\") pod \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\" (UID: \"81e92a80-24f9-4a24-b6b2-5e53b21879b2\") " Nov 02 17:51:57.805158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.794577 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-openshift-ca" pod "81e92a80-24f9-4a24-b6b2-5e53b21879b2" (UID: "81e92a80-24f9-4a24-b6b2-5e53b21879b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:57.847120 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.841199 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-xks-ca" pod "81e92a80-24f9-4a24-b6b2-5e53b21879b2" (UID: "81e92a80-24f9-4a24-b6b2-5e53b21879b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:51:57.883101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.878156 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:57.883101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.878189 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e92a80-24f9-4a24-b6b2-5e53b21879b2-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:57.932143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.929895 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e92a80-24f9-4a24-b6b2-5e53b21879b2-kube-api-access-khdm2" pod "81e92a80-24f9-4a24-b6b2-5e53b21879b2" (UID: "81e92a80-24f9-4a24-b6b2-5e53b21879b2"). InnerVolumeSpecName "kube-api-access-khdm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:57.987998 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:57.986466 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khdm2\" (UniqueName: \"kubernetes.io/projected/81e92a80-24f9-4a24-b6b2-5e53b21879b2-kube-api-access-khdm2\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:51:58.019196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.009218 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:51:58.019196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.009285 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:51:58.027168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.023626 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:51:58.027168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.023703 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 17:51:58.109583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.109153 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:58.112110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.111417 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:58.112312 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.112293 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" Nov 02 17:51:58.130446 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.130420 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" Nov 02 17:51:58.130695 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.130444 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b" event={"ID":"81e92a80-24f9-4a24-b6b2-5e53b21879b2","Type":"ContainerDied","Data":"f46522694000ee1042755e319356233d8491d20636bb61baeac50683fc5bc1fa"} Nov 02 17:51:58.130695 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.130487 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46522694000ee1042755e319356233d8491d20636bb61baeac50683fc5bc1fa" Nov 02 17:51:58.728118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.725169 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:51:58.728118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.725228 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:51:58.940732 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:58.940685 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:51:59.131307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:59.131210 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:51:59.131899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:59.131860 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 17:51:59.742107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:59.741441 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:51:59.742107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:51:59.741525 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:52:00.028166 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:00.027759 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:52:00.028166 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:00.027831 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 17:52:00.706054 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:00.705982 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:00.735049 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:00.735008 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:52:00.735384 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:00.735354 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:52:00.840449 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:00.840413 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:01.023116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:01.021557 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 17:52:01.023116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:01.021619 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 17:52:01.177134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:01.172245 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:01.738491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:01.734149 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:52:01.738491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:01.734222 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 17:52:02.267799 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:02.267150 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:52:02.268176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:02.268117 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:52:02.272397 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:02.272343 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:52:02.460510 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:02.460470 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:03.232770 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.232735 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz"] Nov 02 17:52:03.233997 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.233972 4310 cpu_manager.go:414] "containerMap: removing container" podUID="81e92a80-24f9-4a24-b6b2-5e53b21879b2" containerName="revoke-keys" Nov 02 17:52:03.234161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.234150 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e92a80-24f9-4a24-b6b2-5e53b21879b2" containerName="revoke-keys" Nov 02 17:52:03.234391 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.234381 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e92a80-24f9-4a24-b6b2-5e53b21879b2" containerName="revoke-keys" Nov 02 17:52:03.243166 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.243139 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.258071 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.258042 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea752f\"/\"default-dockercfg-wm8ft\"" Nov 02 17:52:03.266386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.266356 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea752f\"/\"distinct-w4-ea752f-kserve-self-signed-certs\"" Nov 02 17:52:03.267831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.267807 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea752f\"/\"kube-root-ca.crt\"" Nov 02 17:52:03.268058 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.268042 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea752f\"/\"openshift-service-ca.crt\"" Nov 02 17:52:03.328946 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.328916 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz"] Nov 02 17:52:03.334100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.333364 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:03.434548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.413673 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-home\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.434913 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.434885 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-model-cache\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.435269 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.435249 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tmp-dir\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.435450 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.435434 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tls-certs\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.435641 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.435624 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt2vl\" (UniqueName: \"kubernetes.io/projected/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-kube-api-access-rt2vl\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.435769 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.435754 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-dshm\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.546028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.537958 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tls-certs\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.546028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.538072 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt2vl\" (UniqueName: \"kubernetes.io/projected/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-kube-api-access-rt2vl\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.546028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.538127 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-dshm\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.546028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.538192 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-home\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.546028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.538242 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-model-cache\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.546028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.538407 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tmp-dir\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.546028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.538891 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tmp-dir\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.546028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.540868 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-home\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.546028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.541720 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-model-cache\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.565552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.565470 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-dshm\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.584595 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.584510 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tls-certs\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.591162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.587360 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt2vl\" (UniqueName: \"kubernetes.io/projected/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-kube-api-access-rt2vl\") pod \"distinct-w4-ea752f-kserve-66d58667cf-kjlmz\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.606023 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:52:03.605992 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:52:03.629210 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.629169 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:03.724962 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.715272 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:52:03.724962 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.720592 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:52:03.729617 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.729583 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:03.729783 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.729662 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:52:03.921354 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.921280 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp"] Nov 02 17:52:03.926621 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.926600 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:03.956861 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.948483 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:52:03.956861 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.949624 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:52:03.957898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.957870 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp"] Nov 02 17:52:03.963645 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:03.963622 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz"] Nov 02 17:52:04.053391 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.053360 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5c5a11-v9zdp\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:04.053565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.053483 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqt56\" (UniqueName: \"kubernetes.io/projected/318ca266-1cfd-451a-9e7c-b2145e9c512c-kube-api-access-kqt56\") pod \"maas-api-revoke-sub-2a5c5a11-v9zdp\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:04.053652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.053563 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5c5a11-v9zdp\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:04.155291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.155258 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5c5a11-v9zdp\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:04.155469 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.155347 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqt56\" (UniqueName: \"kubernetes.io/projected/318ca266-1cfd-451a-9e7c-b2145e9c512c-kube-api-access-kqt56\") pod \"maas-api-revoke-sub-2a5c5a11-v9zdp\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:04.155469 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.155380 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5c5a11-v9zdp\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:04.156034 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.156011 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5c5a11-v9zdp\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:04.157722 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.157701 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5c5a11-v9zdp\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:04.165480 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.165458 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqt56\" (UniqueName: \"kubernetes.io/projected/318ca266-1cfd-451a-9e7c-b2145e9c512c-kube-api-access-kqt56\") pod \"maas-api-revoke-sub-2a5c5a11-v9zdp\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:04.269726 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.269695 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:04.308424 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.308390 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz"] Nov 02 17:52:04.313857 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.313828 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:04.337434 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.337404 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" event={"ID":"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8","Type":"ContainerStarted","Data":"3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2"} Nov 02 17:52:04.337623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.337604 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" event={"ID":"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8","Type":"ContainerStarted","Data":"27afaf03e901ac4173d071af5fd9094d556071493d9c8c93b7de27f9eaa5528c"} Nov 02 17:52:04.357061 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.357029 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz"] Nov 02 17:52:04.357868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.357842 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2438355-2twrz\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:04.357993 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.357976 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2438355-2twrz\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:04.358153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.358136 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvf64\" (UniqueName: \"kubernetes.io/projected/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-kube-api-access-bvf64\") pod \"maas-api-revoke-sub-b2438355-2twrz\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:04.459450 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.459412 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvf64\" (UniqueName: \"kubernetes.io/projected/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-kube-api-access-bvf64\") pod \"maas-api-revoke-sub-b2438355-2twrz\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:04.459741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.459649 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2438355-2twrz\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:04.460338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.460101 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2438355-2twrz\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:04.460851 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.460828 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2438355-2twrz\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:04.462586 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.462563 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2438355-2twrz\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:04.484712 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.484681 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvf64\" (UniqueName: \"kubernetes.io/projected/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-kube-api-access-bvf64\") pod \"maas-api-revoke-sub-b2438355-2twrz\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:04.493230 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.493184 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podStartSLOduration=1.493170858 podStartE2EDuration="1.493170858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:04.49036608 +0000 UTC m=+721.933448716" watchObservedRunningTime="2026-11-02 17:52:04.493170858 +0000 UTC m=+721.936253520" Nov 02 17:52:04.556959 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.556933 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp"] Nov 02 17:52:04.557392 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:04.557358 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318ca266_1cfd_451a_9e7c_b2145e9c512c.slice/crio-3ea180222aece1485892be7e203bcdfd852c7e6f687455165c1d46fe495775b3 WatchSource:0}: Error finding container 3ea180222aece1485892be7e203bcdfd852c7e6f687455165c1d46fe495775b3: Status 404 returned error can't find the container with id 3ea180222aece1485892be7e203bcdfd852c7e6f687455165c1d46fe495775b3 Nov 02 17:52:04.643519 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.643250 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:04.648115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.647648 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:04.927834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:04.927801 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:05.024005 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:05.023981 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc882aa_7cbf_4d3a_85eb_5d6459f787ab.slice/crio-2a6787153a812243578dd9333a4d86510bf4ac771bd3cd1288c658667e4d5879 WatchSource:0}: Error finding container 2a6787153a812243578dd9333a4d86510bf4ac771bd3cd1288c658667e4d5879: Status 404 returned error can't find the container with id 2a6787153a812243578dd9333a4d86510bf4ac771bd3cd1288c658667e4d5879 Nov 02 17:52:05.027105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:05.027068 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz"] Nov 02 17:52:05.344120 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:05.344066 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" event={"ID":"318ca266-1cfd-451a-9e7c-b2145e9c512c","Type":"ContainerStarted","Data":"58ae626fb4e7df3a72e5ec9477870dde5283f99ff6e5c4747166f869930827c9"} Nov 02 17:52:05.344530 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:05.344171 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" event={"ID":"318ca266-1cfd-451a-9e7c-b2145e9c512c","Type":"ContainerStarted","Data":"3ea180222aece1485892be7e203bcdfd852c7e6f687455165c1d46fe495775b3"} Nov 02 17:52:05.345603 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:05.345572 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" event={"ID":"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab","Type":"ContainerStarted","Data":"2a6787153a812243578dd9333a4d86510bf4ac771bd3cd1288c658667e4d5879"} Nov 02 17:52:05.369399 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:05.369349 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" podStartSLOduration=2.369335029 podStartE2EDuration="2.369335029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:05.366311267 +0000 UTC m=+722.809393904" watchObservedRunningTime="2026-11-02 17:52:05.369335029 +0000 UTC m=+722.812417706" Nov 02 17:52:05.708513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:05.708474 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:06.353512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:06.353477 4310 generic.go:356] "Generic (PLEG): container finished" podID="dfc882aa-7cbf-4d3a-85eb-5d6459f787ab" containerID="be8536cd41f6c4e492cba5011ef87c72fea8ace12466b1b05c2d84b9c7d3a240" exitCode=0 Nov 02 17:52:06.353966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:06.353565 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" event={"ID":"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab","Type":"ContainerDied","Data":"be8536cd41f6c4e492cba5011ef87c72fea8ace12466b1b05c2d84b9c7d3a240"} Nov 02 17:52:06.355205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:06.355184 4310 generic.go:356] "Generic (PLEG): container finished" podID="318ca266-1cfd-451a-9e7c-b2145e9c512c" containerID="58ae626fb4e7df3a72e5ec9477870dde5283f99ff6e5c4747166f869930827c9" exitCode=0 Nov 02 17:52:06.355345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:06.355267 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" event={"ID":"318ca266-1cfd-451a-9e7c-b2145e9c512c","Type":"ContainerDied","Data":"58ae626fb4e7df3a72e5ec9477870dde5283f99ff6e5c4747166f869930827c9"} Nov 02 17:52:06.736451 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:06.735972 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:06.917887 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:06.917853 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf"] Nov 02 17:52:06.922499 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:06.922478 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:06.949599 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:06.949570 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf"] Nov 02 17:52:07.017713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.017629 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74rpb\" (UniqueName: \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-kube-api-access-74rpb\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.017861 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.017740 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.017861 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.017768 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.017936 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.017867 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.017968 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.017948 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.018001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.017987 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.018120 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.018107 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a972210a-5a66-4012-9cb4-20120b1b3c21-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.018161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.018131 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.018208 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.018194 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119012 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a972210a-5a66-4012-9cb4-20120b1b3c21-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119055 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119138 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119168 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74rpb\" (UniqueName: \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-kube-api-access-74rpb\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119246 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119486 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119270 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119486 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119338 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119486 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119414 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119644 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119529 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119698 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119643 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119897 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.119916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.119905 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a972210a-5a66-4012-9cb4-20120b1b3c21-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.120067 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.120014 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.120067 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.120061 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.121733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.121712 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.121851 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.121834 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.131478 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.131445 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.131702 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.131687 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74rpb\" (UniqueName: \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-kube-api-access-74rpb\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.180058 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.180025 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4"] Nov 02 17:52:07.185448 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.185429 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.204364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.204335 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4"] Nov 02 17:52:07.249338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.249299 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:07.325111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.325041 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153da82d-3725-4598-b171-e8b6b4205bf3-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.325284 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.325265 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.325510 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.325418 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.325510 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.325469 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.325641 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.325554 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.325641 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.325622 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.325757 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.325668 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.325757 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.325719 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lvqn\" (UniqueName: \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-kube-api-access-2lvqn\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.325849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.325755 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153da82d-3725-4598-b171-e8b6b4205bf3-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.427420 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.427373 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.427896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.427460 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.427896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.427490 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.427896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.427520 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.427896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.427576 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.427896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.427615 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.427896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.427664 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lvqn\" (UniqueName: \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-kube-api-access-2lvqn\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.427896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.427701 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153da82d-3725-4598-b171-e8b6b4205bf3-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.427896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.427842 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153da82d-3725-4598-b171-e8b6b4205bf3-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.428931 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.428906 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.429195 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.428982 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.429309 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.429229 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.430282 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.430224 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153da82d-3725-4598-b171-e8b6b4205bf3-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.430733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.430714 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.431461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.431270 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:07.432258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.432219 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.435306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.435263 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153da82d-3725-4598-b171-e8b6b4205bf3-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.439538 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.439520 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lvqn\" (UniqueName: \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-kube-api-access-2lvqn\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.441864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.441825 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.577308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.562317 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:07.649999 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.649857 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf"] Nov 02 17:52:07.665437 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:07.665400 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda972210a_5a66_4012_9cb4_20120b1b3c21.slice/crio-36fd146327be73a2a2768a7a0e009b40cdc4bc44567f38663a73e41b7d341c75 WatchSource:0}: Error finding container 36fd146327be73a2a2768a7a0e009b40cdc4bc44567f38663a73e41b7d341c75: Status 404 returned error can't find the container with id 36fd146327be73a2a2768a7a0e009b40cdc4bc44567f38663a73e41b7d341c75 Nov 02 17:52:07.672924 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.672715 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:07.672924 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.672871 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:07.673109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.672970 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:07.692108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.691745 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:07.715518 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.715493 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:07.765686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.765635 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-xks-ca\") pod \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " Nov 02 17:52:07.765805 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.765744 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-openshift-ca\") pod \"318ca266-1cfd-451a-9e7c-b2145e9c512c\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " Nov 02 17:52:07.766175 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.766145 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-openshift-ca" pod "318ca266-1cfd-451a-9e7c-b2145e9c512c" (UID: "318ca266-1cfd-451a-9e7c-b2145e9c512c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:07.766274 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.766201 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-kube-api-access-bvf64\" (UniqueName: \"kubernetes.io/projected/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-kube-api-access-bvf64\") pod \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " Nov 02 17:52:07.766370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.766350 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318ca266-1cfd-451a-9e7c-b2145e9c512c-kube-api-access-kqt56\" (UniqueName: \"kubernetes.io/projected/318ca266-1cfd-451a-9e7c-b2145e9c512c-kube-api-access-kqt56\") pod \"318ca266-1cfd-451a-9e7c-b2145e9c512c\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " Nov 02 17:52:07.766718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.766696 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-openshift-ca\") pod \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\" (UID: \"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab\") " Nov 02 17:52:07.766833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.766816 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-xks-ca\") pod \"318ca266-1cfd-451a-9e7c-b2145e9c512c\" (UID: \"318ca266-1cfd-451a-9e7c-b2145e9c512c\") " Nov 02 17:52:07.767214 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.767186 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-openshift-ca" pod "dfc882aa-7cbf-4d3a-85eb-5d6459f787ab" (UID: "dfc882aa-7cbf-4d3a-85eb-5d6459f787ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:07.767706 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.767679 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:07.767816 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.767794 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:07.770889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.770459 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-kube-api-access-bvf64" pod "dfc882aa-7cbf-4d3a-85eb-5d6459f787ab" (UID: "dfc882aa-7cbf-4d3a-85eb-5d6459f787ab"). InnerVolumeSpecName "kube-api-access-bvf64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:07.772312 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.771371 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-xks-ca" pod "dfc882aa-7cbf-4d3a-85eb-5d6459f787ab" (UID: "dfc882aa-7cbf-4d3a-85eb-5d6459f787ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:07.772312 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.771378 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318ca266-1cfd-451a-9e7c-b2145e9c512c-kube-api-access-kqt56" pod "318ca266-1cfd-451a-9e7c-b2145e9c512c" (UID: "318ca266-1cfd-451a-9e7c-b2145e9c512c"). InnerVolumeSpecName "kube-api-access-kqt56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:07.772592 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.772550 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-xks-ca" pod "318ca266-1cfd-451a-9e7c-b2145e9c512c" (UID: "318ca266-1cfd-451a-9e7c-b2145e9c512c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:07.869438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.869349 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:07.869438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.869392 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvf64\" (UniqueName: \"kubernetes.io/projected/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab-kube-api-access-bvf64\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:07.869438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.869409 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqt56\" (UniqueName: \"kubernetes.io/projected/318ca266-1cfd-451a-9e7c-b2145e9c512c-kube-api-access-kqt56\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:07.869438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:07.869425 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/318ca266-1cfd-451a-9e7c-b2145e9c512c-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:08.029322 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.029286 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4"] Nov 02 17:52:08.029769 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:08.029740 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153da82d_3725_4598_b171_e8b6b4205bf3.slice/crio-94b6e11cfe8779b46ac5abc5118c953de9b24ee468d08c99c96cbf827e287ded WatchSource:0}: Error finding container 94b6e11cfe8779b46ac5abc5118c953de9b24ee468d08c99c96cbf827e287ded: Status 404 returned error can't find the container with id 94b6e11cfe8779b46ac5abc5118c953de9b24ee468d08c99c96cbf827e287ded Nov 02 17:52:08.032195 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.032164 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:08.032301 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.032244 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:08.032301 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.032293 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:08.163061 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.163017 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:08.378907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.378877 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" Nov 02 17:52:08.379339 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.378896 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz" event={"ID":"dfc882aa-7cbf-4d3a-85eb-5d6459f787ab","Type":"ContainerDied","Data":"2a6787153a812243578dd9333a4d86510bf4ac771bd3cd1288c658667e4d5879"} Nov 02 17:52:08.379461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.379371 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a6787153a812243578dd9333a4d86510bf4ac771bd3cd1288c658667e4d5879" Nov 02 17:52:08.380947 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.380921 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" Nov 02 17:52:08.380947 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.380941 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp" event={"ID":"318ca266-1cfd-451a-9e7c-b2145e9c512c","Type":"ContainerDied","Data":"3ea180222aece1485892be7e203bcdfd852c7e6f687455165c1d46fe495775b3"} Nov 02 17:52:08.381177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.380983 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ea180222aece1485892be7e203bcdfd852c7e6f687455165c1d46fe495775b3" Nov 02 17:52:08.384119 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.383210 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" event={"ID":"a972210a-5a66-4012-9cb4-20120b1b3c21","Type":"ContainerStarted","Data":"d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0"} Nov 02 17:52:08.384119 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.383266 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" event={"ID":"a972210a-5a66-4012-9cb4-20120b1b3c21","Type":"ContainerStarted","Data":"36fd146327be73a2a2768a7a0e009b40cdc4bc44567f38663a73e41b7d341c75"} Nov 02 17:52:08.386155 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.386127 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" event={"ID":"153da82d-3725-4598-b171-e8b6b4205bf3","Type":"ContainerStarted","Data":"b9db4b5dc82ba36d6f7a85bfe64a0d183b3b75cc1021971db5699b96d2306d06"} Nov 02 17:52:08.386251 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.386177 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" event={"ID":"153da82d-3725-4598-b171-e8b6b4205bf3","Type":"ContainerStarted","Data":"94b6e11cfe8779b46ac5abc5118c953de9b24ee468d08c99c96cbf827e287ded"} Nov 02 17:52:08.416033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.415980 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" podStartSLOduration=2.4159650409999998 podStartE2EDuration="2.415965041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:08.412278151 +0000 UTC m=+725.855360828" watchObservedRunningTime="2026-11-02 17:52:08.415965041 +0000 UTC m=+725.859047677" Nov 02 17:52:08.465452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.461374 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" podStartSLOduration=1.46135152 podStartE2EDuration="1.46135152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:08.45242424 +0000 UTC m=+725.895506877" watchObservedRunningTime="2026-11-02 17:52:08.46135152 +0000 UTC m=+725.904434157" Nov 02 17:52:08.588310 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:08.588276 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:09.254103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:09.250470 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:09.271833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:09.271791 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:52:09.273423 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:09.273392 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:52:09.273933 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:09.273133 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:09.592195 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:09.591257 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:52:09.592195 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:09.591319 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:52:10.263934 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.260544 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:10.263934 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.261190 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:52:10.263934 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.261240 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:52:10.588688 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.588598 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:52:10.588943 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.588917 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:52:10.727102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.727062 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh"] Nov 02 17:52:10.728092 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.728060 4310 cpu_manager.go:414] "containerMap: removing container" podUID="dfc882aa-7cbf-4d3a-85eb-5d6459f787ab" containerName="revoke-keys" Nov 02 17:52:10.728201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.728190 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc882aa-7cbf-4d3a-85eb-5d6459f787ab" containerName="revoke-keys" Nov 02 17:52:10.728267 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.728260 4310 cpu_manager.go:414] "containerMap: removing container" podUID="318ca266-1cfd-451a-9e7c-b2145e9c512c" containerName="revoke-keys" Nov 02 17:52:10.728317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.728311 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318ca266-1cfd-451a-9e7c-b2145e9c512c" containerName="revoke-keys" Nov 02 17:52:10.728497 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.728489 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc882aa-7cbf-4d3a-85eb-5d6459f787ab" containerName="revoke-keys" Nov 02 17:52:10.728548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.728542 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="318ca266-1cfd-451a-9e7c-b2145e9c512c" containerName="revoke-keys" Nov 02 17:52:10.733116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.733099 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:10.816098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.814834 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb4ecb\"/\"distinct-2-w2-bb4ecb-kserve-self-signed-certs\"" Nov 02 17:52:10.827127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.819872 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh"] Nov 02 17:52:10.838841 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.838765 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:10.927717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.901255 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd26314-4dca-4ac0-9cee-25efded09e70-tls-certs\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:10.927717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.901360 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-home\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:10.927717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.901567 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-model-cache\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:10.927717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.901597 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-tmp-dir\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:10.927717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.901645 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-dshm\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:10.927717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:10.901671 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbn2s\" (UniqueName: \"kubernetes.io/projected/dbd26314-4dca-4ac0-9cee-25efded09e70-kube-api-access-dbn2s\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.005055 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.005010 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-model-cache\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.025318 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.025276 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-tmp-dir\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.026103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.026065 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-dshm\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.026643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.026621 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbn2s\" (UniqueName: \"kubernetes.io/projected/dbd26314-4dca-4ac0-9cee-25efded09e70-kube-api-access-dbn2s\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.027207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.027187 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd26314-4dca-4ac0-9cee-25efded09e70-tls-certs\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.027905 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.027884 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-home\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.028581 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.005691 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-model-cache\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.028750 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.025972 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-tmp-dir\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.028911 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.028332 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-home\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.054890 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.054857 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd26314-4dca-4ac0-9cee-25efded09e70-tls-certs\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.074744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.074709 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbn2s\" (UniqueName: \"kubernetes.io/projected/dbd26314-4dca-4ac0-9cee-25efded09e70-kube-api-access-dbn2s\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.075370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.075340 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-dshm\") pod \"distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.126441 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.126344 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:11.256101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.255562 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:52:11.256101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.255627 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:52:11.534061 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.533927 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh"] Nov 02 17:52:11.539950 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.539057 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:11.603596 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.603556 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:52:11.605244 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:11.605194 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:52:12.250878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:12.250446 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:52:12.250878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:12.250509 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:52:12.512314 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:12.512224 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" event={"ID":"dbd26314-4dca-4ac0-9cee-25efded09e70","Type":"ContainerStarted","Data":"4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055"} Nov 02 17:52:12.512314 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:12.512268 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" event={"ID":"dbd26314-4dca-4ac0-9cee-25efded09e70","Type":"ContainerStarted","Data":"7897f1183a7246caa14c0da6c2b714c5fa3cf33da2116a1ef70e575cf475eabb"} Nov 02 17:52:12.540790 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:12.540733 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podStartSLOduration=2.540719015 podStartE2EDuration="2.540719015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:12.538330596 +0000 UTC m=+729.981413287" watchObservedRunningTime="2026-11-02 17:52:12.540719015 +0000 UTC m=+729.983801651" Nov 02 17:52:12.591100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:12.590507 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:52:12.591100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:12.590611 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:52:12.690110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:12.680932 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:13.128163 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:13.128134 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:13.257568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:13.257528 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:52:13.258222 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:13.258186 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:52:13.565105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:13.564574 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:13.591750 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:13.591709 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:52:13.592003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:13.591977 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:52:13.653191 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:13.653152 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:13.824577 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:13.824495 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:14.253578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:14.253537 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 17:52:14.253856 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:14.253830 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 17:52:14.549484 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:14.549372 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:14.588776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:14.588739 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 17:52:14.588952 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:14.588815 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 17:52:14.645688 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:14.645655 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:15.256243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:15.256206 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:15.256646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:15.256619 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:15.257521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:15.257497 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:52:15.611826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:15.611735 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:15.612347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:15.612200 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:15.613838 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:15.613810 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:52:15.878274 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:15.878180 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:16.773710 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:16.773643 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:17.624153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:17.624111 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:18.672579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.672544 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:18.697208 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.697177 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m"] Nov 02 17:52:18.701798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.701772 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.721927 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.721899 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ea3867\"/\"distinct-2-w5-ea3867-kserve-self-signed-certs\"" Nov 02 17:52:18.726280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.726252 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m"] Nov 02 17:52:18.814858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.814824 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:18.815292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.815268 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-model-cache\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.815372 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.815303 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tls-certs\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.815372 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.815320 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdc5\" (UniqueName: \"kubernetes.io/projected/1bdb0ab7-10bd-4211-b99a-0cc15697684e-kube-api-access-vhdc5\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.815462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.815441 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-dshm\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.815570 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.815556 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tmp-dir\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.815615 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.815592 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-home\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.920099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.918907 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tmp-dir\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.920099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.918998 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-home\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.920099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.919180 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-model-cache\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.920099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.919213 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tls-certs\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.920099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.919240 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdc5\" (UniqueName: \"kubernetes.io/projected/1bdb0ab7-10bd-4211-b99a-0cc15697684e-kube-api-access-vhdc5\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.920099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.919292 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-dshm\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.920551 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.920294 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-model-cache\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.920610 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.920556 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tmp-dir\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.920819 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.920796 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-home\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.938143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.936926 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-dshm\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.949306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.948962 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tls-certs\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:18.958100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:18.950717 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdc5\" (UniqueName: \"kubernetes.io/projected/1bdb0ab7-10bd-4211-b99a-0cc15697684e-kube-api-access-vhdc5\") pod \"distinct-2-w5-ea3867-kserve-894d86c7-zms5m\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:19.018697 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:19.018391 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:20.401327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:20.401281 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m"] Nov 02 17:52:20.472972 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:20.472931 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bdb0ab7_10bd_4211_b99a_0cc15697684e.slice/crio-8e97838e7e25fd7f8bd496adca57b8f40937e97bc0eb6283e085aaa5db01ddea WatchSource:0}: Error finding container 8e97838e7e25fd7f8bd496adca57b8f40937e97bc0eb6283e085aaa5db01ddea: Status 404 returned error can't find the container with id 8e97838e7e25fd7f8bd496adca57b8f40937e97bc0eb6283e085aaa5db01ddea Nov 02 17:52:20.582126 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:20.537296 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:20.834137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:20.829173 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:20.932705 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:20.932661 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:21.005909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:21.005799 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" event={"ID":"1bdb0ab7-10bd-4211-b99a-0cc15697684e","Type":"ContainerStarted","Data":"8e97838e7e25fd7f8bd496adca57b8f40937e97bc0eb6283e085aaa5db01ddea"} Nov 02 17:52:21.169103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:21.165224 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:21.233099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:21.226827 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:21.961871 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:21.961831 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9"] Nov 02 17:52:21.966832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:21.966804 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:21.986513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:21.986484 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9"] Nov 02 17:52:22.005547 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.005501 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4140a511-0bf5-413c-93d5-f35e2a070523-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.005766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.005746 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.005825 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.005783 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg8k5\" (UniqueName: \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-kube-api-access-qg8k5\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.005950 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.005921 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4140a511-0bf5-413c-93d5-f35e2a070523-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.006072 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.005987 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.006072 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.006066 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.006224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.006161 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.006224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.006212 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.006295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.006234 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.015219 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.015186 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" event={"ID":"1bdb0ab7-10bd-4211-b99a-0cc15697684e","Type":"ContainerStarted","Data":"7283f8e404ba2ee70ae19f9d422844c8b180475ccacaa2094d0ddbca17ba086c"} Nov 02 17:52:22.019237 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.019196 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:22.020815 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.020791 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:22.029446 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.029426 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:52:22.070589 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.070529 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podStartSLOduration=4.070513244 podStartE2EDuration="4.070513244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:22.068488748 +0000 UTC m=+739.511571408" watchObservedRunningTime="2026-11-02 17:52:22.070513244 +0000 UTC m=+739.513595879" Nov 02 17:52:22.134101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.131184 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4140a511-0bf5-413c-93d5-f35e2a070523-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.134101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.131295 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.134101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.131325 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg8k5\" (UniqueName: \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-kube-api-access-qg8k5\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.134101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.131391 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4140a511-0bf5-413c-93d5-f35e2a070523-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.134101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.131453 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.134101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.131493 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.134101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.131590 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.134101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.131674 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.134101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.131700 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.141097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.138252 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-data\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.141097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.138821 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-credential-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.141097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.139042 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-socket\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.141097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.139229 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-certs\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.141097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.139869 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4140a511-0bf5-413c-93d5-f35e2a070523-istiod-ca-cert\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.148016 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.147964 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-istio-token\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.164222 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.164190 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-envoy\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.164788 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.164768 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4140a511-0bf5-413c-93d5-f35e2a070523-istio-podinfo\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.166524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.166472 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg8k5\" (UniqueName: \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-kube-api-access-qg8k5\") pod \"e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.295547 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.285424 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:22.536520 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.536485 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz"] Nov 02 17:52:22.541901 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.541879 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.572725 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.572692 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz"] Nov 02 17:52:22.616508 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:22.616471 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4140a511_0bf5_413c_93d5_f35e2a070523.slice/crio-2f1ab95d50c7132fbb26fd97e83bf4951a7de130ee86ca0b0693e771c61baf9a WatchSource:0}: Error finding container 2f1ab95d50c7132fbb26fd97e83bf4951a7de130ee86ca0b0693e771c61baf9a: Status 404 returned error can't find the container with id 2f1ab95d50c7132fbb26fd97e83bf4951a7de130ee86ca0b0693e771c61baf9a Nov 02 17:52:22.618496 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.618382 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9"] Nov 02 17:52:22.619853 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.619822 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:22.619927 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.619884 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:22.619927 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.619917 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:22.645245 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.645217 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.645378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.645334 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.645378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.645368 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.645468 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.645402 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.645603 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.645579 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlddz\" (UniqueName: \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-kube-api-access-hlddz\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.645651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.645623 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.645732 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.645717 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.645767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.645750 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.645849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.645835 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.747483 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.747429 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.747678 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.747503 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.747678 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.747530 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.747678 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.747561 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.747678 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.747657 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlddz\" (UniqueName: \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-kube-api-access-hlddz\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.747883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.747685 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.747883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.747739 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.747883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.747769 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.747883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.747826 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.748467 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.748425 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.748632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.748609 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.748977 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.748951 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.749266 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.749244 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.749341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.749266 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.751105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.751065 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.751609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.751589 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.763093 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.760209 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlddz\" (UniqueName: \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-kube-api-access-hlddz\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.763093 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.761554 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.857272 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.857183 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:22.913947 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:22.913906 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:23.075192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:23.075150 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" event={"ID":"4140a511-0bf5-413c-93d5-f35e2a070523","Type":"ContainerStarted","Data":"195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72"} Nov 02 17:52:23.075657 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:23.075221 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" event={"ID":"4140a511-0bf5-413c-93d5-f35e2a070523","Type":"ContainerStarted","Data":"2f1ab95d50c7132fbb26fd97e83bf4951a7de130ee86ca0b0693e771c61baf9a"} Nov 02 17:52:23.107531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:23.107414 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" podStartSLOduration=2.107399645 podStartE2EDuration="2.107399645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:23.102237936 +0000 UTC m=+740.545320572" watchObservedRunningTime="2026-11-02 17:52:23.107399645 +0000 UTC m=+740.550482291" Nov 02 17:52:23.152115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:23.152056 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz"] Nov 02 17:52:23.152225 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:23.152055 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23349c1_0d41_41eb_a39a_5e7c1f034b5a.slice/crio-2f919a0f2a1b96068ce79d45459545215fb8ae6e9e8ea129be0d0e40f6d1953b WatchSource:0}: Error finding container 2f919a0f2a1b96068ce79d45459545215fb8ae6e9e8ea129be0d0e40f6d1953b: Status 404 returned error can't find the container with id 2f919a0f2a1b96068ce79d45459545215fb8ae6e9e8ea129be0d0e40f6d1953b Nov 02 17:52:23.154570 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:23.154540 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:23.154699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:23.154621 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:23.154699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:23.154665 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:23.297139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:23.293359 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:23.298271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:23.298229 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:52:23.298404 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:23.298304 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:52:23.820635 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:23.820595 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:24.102320 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:24.102226 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" event={"ID":"d23349c1-0d41-41eb-a39a-5e7c1f034b5a","Type":"ContainerStarted","Data":"dd4e4a1680024ff19d22cedf70a187a33d07dced10cd875cb494079d2d7a98c2"} Nov 02 17:52:24.102320 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:24.102285 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" event={"ID":"d23349c1-0d41-41eb-a39a-5e7c1f034b5a","Type":"ContainerStarted","Data":"2f919a0f2a1b96068ce79d45459545215fb8ae6e9e8ea129be0d0e40f6d1953b"} Nov 02 17:52:24.174101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:24.152360 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" podStartSLOduration=2.15234032 podStartE2EDuration="2.15234032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:24.150595912 +0000 UTC m=+741.593678562" watchObservedRunningTime="2026-11-02 17:52:24.15234032 +0000 UTC m=+741.595422957" Nov 02 17:52:24.293103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:24.292713 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:52:24.293103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:24.292786 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:52:24.864188 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:24.864144 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:24.871236 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:24.871198 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:52:24.871470 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:24.871442 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 17:52:24.994644 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:24.994606 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:25.292627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:25.292590 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:52:25.293256 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:25.293207 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:52:25.862102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:25.861518 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:52:25.862102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:25.861581 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 17:52:26.112027 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:26.111986 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:26.294638 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:26.294593 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:52:26.295332 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:26.295295 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:52:26.863103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:26.859484 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:52:26.863103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:26.859546 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 17:52:27.245102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:27.241221 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:27.296396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:27.296350 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:52:27.296986 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:27.296952 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:52:27.864682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:27.864648 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:52:27.864938 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:27.864912 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 17:52:28.211661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:28.211573 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:28.292560 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:28.292526 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:52:28.292872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:28.292843 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:52:28.857786 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:28.857744 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:52:28.858446 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:28.858411 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 17:52:29.019251 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:29.019218 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:29.033000 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:29.032965 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:29.116994 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:29.116906 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:29.292666 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:29.292623 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 02 17:52:29.292833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:29.292684 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 02 17:52:29.477916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:29.477881 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:29.857646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:29.857565 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:52:29.857865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:29.857705 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:29.858305 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:29.857840 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 17:52:30.294333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.294292 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:30.364844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.364796 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:30.366174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.366149 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:30.367668 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.367645 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:52:30.749120 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.749068 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv"] Nov 02 17:52:30.760367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.760338 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv"] Nov 02 17:52:30.761982 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.760561 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:30.771375 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.769444 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:52:30.771375 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.769510 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:52:30.965138 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.951363 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fe38828-924qv\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:30.979353 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.965860 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fe38828-924qv\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:30.979839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:30.979804 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hks5\" (UniqueName: \"kubernetes.io/projected/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-kube-api-access-9hks5\") pod \"maas-api-revoke-sub-3fe38828-924qv\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:31.082286 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:31.081608 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hks5\" (UniqueName: \"kubernetes.io/projected/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-kube-api-access-9hks5\") pod \"maas-api-revoke-sub-3fe38828-924qv\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:31.082286 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:31.081762 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fe38828-924qv\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:31.082286 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:31.081909 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fe38828-924qv\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:31.082897 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:31.082848 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fe38828-924qv\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:31.139103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:31.122058 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hks5\" (UniqueName: \"kubernetes.io/projected/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-kube-api-access-9hks5\") pod \"maas-api-revoke-sub-3fe38828-924qv\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:31.167170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:31.160213 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fe38828-924qv\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:31.389303 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:31.388329 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:31.840707 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:31.796305 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:31.840707 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:31.838653 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:31.840707 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:31.840016 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:52:32.314481 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:32.314435 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:32.403532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:32.403437 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv"] Nov 02 17:52:32.551460 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:32.551433 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" event={"ID":"6d24d06a-1de2-46fe-a9a8-a3e63a60640b","Type":"ContainerStarted","Data":"c89c74a7700682506aa8cd7bc28a7e99a27eef0fd5dc877c4faf85e5f1cb404e"} Nov 02 17:52:33.369016 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:33.368977 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:33.546407 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:52:33.546360 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:52:33.563422 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:33.563383 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" event={"ID":"6d24d06a-1de2-46fe-a9a8-a3e63a60640b","Type":"ContainerStarted","Data":"45592a15844cbe96f1b6cfefa2c6e1a9fe60fc4aca77f20d8c52ac720233c25c"} Nov 02 17:52:34.156477 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.156444 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:34.572281 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.572245 4310 generic.go:356] "Generic (PLEG): container finished" podID="6d24d06a-1de2-46fe-a9a8-a3e63a60640b" containerID="45592a15844cbe96f1b6cfefa2c6e1a9fe60fc4aca77f20d8c52ac720233c25c" exitCode=0 Nov 02 17:52:34.572723 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.572337 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" event={"ID":"6d24d06a-1de2-46fe-a9a8-a3e63a60640b","Type":"ContainerDied","Data":"45592a15844cbe96f1b6cfefa2c6e1a9fe60fc4aca77f20d8c52ac720233c25c"} Nov 02 17:52:34.730771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.730746 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:34.811744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.811708 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-kube-api-access-9hks5\" (UniqueName: \"kubernetes.io/projected/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-kube-api-access-9hks5\") pod \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " Nov 02 17:52:34.811914 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.811794 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-openshift-ca\") pod \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " Nov 02 17:52:34.812036 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.812019 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-xks-ca\") pod \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\" (UID: \"6d24d06a-1de2-46fe-a9a8-a3e63a60640b\") " Nov 02 17:52:34.812161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.812137 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-openshift-ca" pod "6d24d06a-1de2-46fe-a9a8-a3e63a60640b" (UID: "6d24d06a-1de2-46fe-a9a8-a3e63a60640b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:34.812522 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.812504 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:34.813971 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.813948 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-kube-api-access-9hks5" pod "6d24d06a-1de2-46fe-a9a8-a3e63a60640b" (UID: "6d24d06a-1de2-46fe-a9a8-a3e63a60640b"). InnerVolumeSpecName "kube-api-access-9hks5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:34.814141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.814125 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-xks-ca" pod "6d24d06a-1de2-46fe-a9a8-a3e63a60640b" (UID: "6d24d06a-1de2-46fe-a9a8-a3e63a60640b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:34.899814 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.899778 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:34.914523 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.914493 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hks5\" (UniqueName: \"kubernetes.io/projected/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-kube-api-access-9hks5\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:34.914523 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:34.914521 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d24d06a-1de2-46fe-a9a8-a3e63a60640b-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:35.265854 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:35.265824 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:35.580200 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:35.580114 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" event={"ID":"6d24d06a-1de2-46fe-a9a8-a3e63a60640b","Type":"ContainerDied","Data":"c89c74a7700682506aa8cd7bc28a7e99a27eef0fd5dc877c4faf85e5f1cb404e"} Nov 02 17:52:35.580200 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:35.580141 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv" Nov 02 17:52:35.580200 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:35.580188 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c89c74a7700682506aa8cd7bc28a7e99a27eef0fd5dc877c4faf85e5f1cb404e" Nov 02 17:52:35.906552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:35.906469 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:36.695536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:36.695504 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:37.181921 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.181880 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm"] Nov 02 17:52:37.182546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.182528 4310 cpu_manager.go:414] "containerMap: removing container" podUID="6d24d06a-1de2-46fe-a9a8-a3e63a60640b" containerName="revoke-keys" Nov 02 17:52:37.182546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.182544 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d24d06a-1de2-46fe-a9a8-a3e63a60640b" containerName="revoke-keys" Nov 02 17:52:37.182716 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.182683 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d24d06a-1de2-46fe-a9a8-a3e63a60640b" containerName="revoke-keys" Nov 02 17:52:37.187340 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.187318 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.207846 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.207819 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm"] Nov 02 17:52:37.265168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.265132 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded09c48-50f3-417f-9502-39403c2878d4-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.265362 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.265214 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.265436 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.265357 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts9bv\" (UniqueName: \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-kube-api-access-ts9bv\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.265436 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.265408 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.265565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.265549 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.265632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.265596 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.265725 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.265701 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded09c48-50f3-417f-9502-39403c2878d4-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.265870 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.265850 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.265971 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.265955 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.366785 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.366748 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.366974 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.366814 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts9bv\" (UniqueName: \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-kube-api-access-ts9bv\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.366974 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.366849 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.366974 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.366923 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.366974 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.366963 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.367256 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.366997 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded09c48-50f3-417f-9502-39403c2878d4-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.367256 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.367200 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.367356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.367300 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.367356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.367346 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded09c48-50f3-417f-9502-39403c2878d4-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.367507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.367479 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.367618 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.367514 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.367618 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.367566 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.367766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.367748 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.368201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.368178 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded09c48-50f3-417f-9502-39403c2878d4-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.369523 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.369503 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.369859 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.369840 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded09c48-50f3-417f-9502-39403c2878d4-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.376697 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.376676 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.376790 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.376720 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts9bv\" (UniqueName: \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-kube-api-access-ts9bv\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.502869 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.502840 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:37.712640 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.712607 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:37.789364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.789279 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn"] Nov 02 17:52:37.795727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.795680 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.803776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.803747 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz"] Nov 02 17:52:37.811493 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.811433 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.818837 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.817778 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn"] Nov 02 17:52:37.823604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.823129 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea752f\"/\"distinct-2-w4-ea752f-kserve-self-signed-certs\"" Nov 02 17:52:37.840109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.840067 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm"] Nov 02 17:52:37.853157 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:37.853127 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded09c48_50f3_417f_9502_39403c2878d4.slice/crio-c0f630756027c9e97b1c3a20ef9413b8518f4b514f48dc716df0a974bc066e2d WatchSource:0}: Error finding container c0f630756027c9e97b1c3a20ef9413b8518f4b514f48dc716df0a974bc066e2d: Status 404 returned error can't find the container with id c0f630756027c9e97b1c3a20ef9413b8518f4b514f48dc716df0a974bc066e2d Nov 02 17:52:37.855640 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.855620 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz"] Nov 02 17:52:37.856608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.856568 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:37.856702 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.856649 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:37.856702 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.856691 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:37.875835 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.875796 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.875976 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.875844 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-dshm\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.875976 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.875917 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-model-cache\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.875976 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.875972 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x22mm\" (UniqueName: \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-kube-api-access-x22mm\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.876218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.876018 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74414d9b-bc66-443d-89b6-03901d5b7890-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.876218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.876045 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.876218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.876103 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ghx8\" (UniqueName: \"kubernetes.io/projected/784056a6-b287-4aa3-af0e-b1175a88200b-kube-api-access-9ghx8\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.876218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.876190 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-tmp-dir\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.876419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.876220 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-home\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.876419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.876258 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.876419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.876282 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.876419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.876318 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784056a6-b287-4aa3-af0e-b1175a88200b-tls-certs\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.876419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.876342 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.876419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.876404 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74414d9b-bc66-443d-89b6-03901d5b7890-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.876754 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.876517 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.902606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.902577 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980320 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x22mm\" (UniqueName: \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-kube-api-access-x22mm\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980402 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74414d9b-bc66-443d-89b6-03901d5b7890-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980433 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980474 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ghx8\" (UniqueName: \"kubernetes.io/projected/784056a6-b287-4aa3-af0e-b1175a88200b-kube-api-access-9ghx8\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980556 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-tmp-dir\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980587 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-home\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980614 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980638 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980674 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784056a6-b287-4aa3-af0e-b1175a88200b-tls-certs\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980697 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980752 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74414d9b-bc66-443d-89b6-03901d5b7890-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980860 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980942 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.980974 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-dshm\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.983345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.981053 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-model-cache\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:37.997661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:37.981567 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-model-cache\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:38.007207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.004722 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-certs\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:38.007207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.004992 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:38.007207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.005730 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74414d9b-bc66-443d-89b6-03901d5b7890-istiod-ca-cert\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:38.007207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.006311 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-home\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:38.007207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.006540 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-credential-socket\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:38.007207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.006783 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-data\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:38.008013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.007978 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-tmp-dir\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:38.008761 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.008742 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74414d9b-bc66-443d-89b6-03901d5b7890-istio-podinfo\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:38.012362 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.012311 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-dshm\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:38.013062 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.013010 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-envoy\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:38.013062 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.013025 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x22mm\" (UniqueName: \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-kube-api-access-x22mm\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:38.025623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.025544 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-istio-token\") pod \"e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:38.028920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.028853 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784056a6-b287-4aa3-af0e-b1175a88200b-tls-certs\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:38.037605 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.037540 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ghx8\" (UniqueName: \"kubernetes.io/projected/784056a6-b287-4aa3-af0e-b1175a88200b-kube-api-access-9ghx8\") pod \"distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:38.149149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.124366 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:38.149149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.132561 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:38.477357 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.477325 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz"] Nov 02 17:52:38.478242 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:38.478194 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod784056a6_b287_4aa3_af0e_b1175a88200b.slice/crio-d54998ccbf55368f4df6b038af3f3d4b08dd61807e1d86b128c427b539543770 WatchSource:0}: Error finding container d54998ccbf55368f4df6b038af3f3d4b08dd61807e1d86b128c427b539543770: Status 404 returned error can't find the container with id d54998ccbf55368f4df6b038af3f3d4b08dd61807e1d86b128c427b539543770 Nov 02 17:52:38.517364 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:38.517324 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74414d9b_bc66_443d_89b6_03901d5b7890.slice/crio-c0a104fb60432d164a29a9fb7c11f42bd698dc819a34ef76b131904771bede67 WatchSource:0}: Error finding container c0a104fb60432d164a29a9fb7c11f42bd698dc819a34ef76b131904771bede67: Status 404 returned error can't find the container with id c0a104fb60432d164a29a9fb7c11f42bd698dc819a34ef76b131904771bede67 Nov 02 17:52:38.520291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.520269 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn"] Nov 02 17:52:38.520788 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.520632 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:38.520903 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.520843 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:38.520992 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.520971 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:38.609022 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.608981 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" event={"ID":"ded09c48-50f3-417f-9502-39403c2878d4","Type":"ContainerStarted","Data":"a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a"} Nov 02 17:52:38.609194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.609047 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" event={"ID":"ded09c48-50f3-417f-9502-39403c2878d4","Type":"ContainerStarted","Data":"c0f630756027c9e97b1c3a20ef9413b8518f4b514f48dc716df0a974bc066e2d"} Nov 02 17:52:38.610606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.610580 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" event={"ID":"74414d9b-bc66-443d-89b6-03901d5b7890","Type":"ContainerStarted","Data":"c0a104fb60432d164a29a9fb7c11f42bd698dc819a34ef76b131904771bede67"} Nov 02 17:52:38.612740 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.612717 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" event={"ID":"784056a6-b287-4aa3-af0e-b1175a88200b","Type":"ContainerStarted","Data":"d54998ccbf55368f4df6b038af3f3d4b08dd61807e1d86b128c427b539543770"} Nov 02 17:52:38.634861 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.634811 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podStartSLOduration=1.634797209 podStartE2EDuration="1.634797209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:38.63196227 +0000 UTC m=+756.075044907" watchObservedRunningTime="2026-11-02 17:52:38.634797209 +0000 UTC m=+756.077879845" Nov 02 17:52:38.866101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:38.859306 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:39.504003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:39.503968 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:39.505679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:39.505651 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 17:52:39.505790 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:39.505696 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 17:52:39.623123 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:39.623061 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" event={"ID":"74414d9b-bc66-443d-89b6-03901d5b7890","Type":"ContainerStarted","Data":"64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc"} Nov 02 17:52:39.625569 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:39.625539 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" event={"ID":"784056a6-b287-4aa3-af0e-b1175a88200b","Type":"ContainerStarted","Data":"3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d"} Nov 02 17:52:39.703966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:39.703903 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" podStartSLOduration=2.7038853510000003 podStartE2EDuration="2.703885351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:39.656435376 +0000 UTC m=+757.099518012" watchObservedRunningTime="2026-11-02 17:52:39.703885351 +0000 UTC m=+757.146967987" Nov 02 17:52:39.704831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:39.704758 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podStartSLOduration=2.704742275 podStartE2EDuration="2.704742275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:39.692617564 +0000 UTC m=+757.135700200" watchObservedRunningTime="2026-11-02 17:52:39.704742275 +0000 UTC m=+757.147824912" Nov 02 17:52:40.126210 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:40.125617 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:40.131711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:40.130769 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:52:40.131711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:40.130828 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 17:52:40.158167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:40.154947 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:40.158167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:40.156507 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:40.503536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:40.503486 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 17:52:40.503753 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:40.503551 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 17:52:41.125732 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:41.125698 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:52:41.125911 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:41.125755 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 17:52:41.229921 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:41.229887 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:41.503958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:41.503903 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 17:52:41.504159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:41.503982 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 17:52:42.125616 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:42.125585 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:52:42.125800 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:42.125643 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 17:52:42.277102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:42.276594 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:42.506103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:42.505481 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 17:52:42.506103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:42.505543 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 17:52:43.125755 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:43.125725 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:52:43.125930 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:43.125773 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 17:52:43.204739 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:43.204702 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:43.503252 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:43.503219 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 17:52:43.503663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:43.503292 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 17:52:44.125991 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:44.125955 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:52:44.126196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:44.126020 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 17:52:44.234480 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:44.234417 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:44.503560 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:44.503518 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 17:52:44.503982 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:44.503587 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 17:52:45.118629 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:45.118590 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:45.131567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:45.131530 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 17:52:45.131845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:45.131815 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 17:52:45.432033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:45.431739 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:45.771769 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:45.771730 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:45.773624 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:45.773599 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:45.774401 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:45.774376 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 17:52:45.906767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:45.906729 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:46.130903 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:46.130819 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:46.131255 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:46.131234 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:46.131965 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:46.131944 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:52:46.313001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:46.312964 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:47.241601 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:47.241558 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:48.133658 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:48.133625 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:48.209180 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:48.204974 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:48.244267 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:48.244225 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:48.770865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:48.770814 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:50.034235 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:50.034182 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded" Nov 02 17:52:50.122105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:50.119979 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:50.418160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:50.412625 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:50.441224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:50.441178 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:52:50.498415 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:50.498377 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:50.541221 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:50.541174 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:52:50.700732 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:50.700626 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:50.772136 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:50.763162 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:52:50.788215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:50.788166 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:51.053103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.043178 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:51.121228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.121178 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:51.122247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.122220 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:52:51.444688 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.444561 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:51.445366 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.445341 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:51.500184 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.500138 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:51.500931 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.500909 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:52:51.560926 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.560787 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:52:51.565508 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.565470 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:51.719158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.691998 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:52:51.719158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.713749 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:51.719158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.716151 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:52:51.759408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.742713 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:52:51.807663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.763580 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:51.807663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.763908 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:51.807663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.798175 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:52:51.807663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.798498 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:52:51.866277 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.862802 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:52:51.885825 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.885797 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:52:51.933590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.933560 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:52:51.937933 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.937897 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:52:51.952874 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:51.952155 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:52:52.298257 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.298225 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:52.379101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.377917 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2"] Nov 02 17:52:52.403105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.402452 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.452001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.451970 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2"] Nov 02 17:52:52.459218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.459186 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs"] Nov 02 17:52:52.464501 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.464474 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.470909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.470884 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs"] Nov 02 17:52:52.538090 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.538043 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.538306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.538195 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.538306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.538243 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.538306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.538276 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.538432 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.538393 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.538432 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.538419 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.538544 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.538529 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.538601 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.538575 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gn2\" (UniqueName: \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-kube-api-access-79gn2\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.538746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.538731 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10d03bf3-ecba-4349-ac6c-0410e573d494-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.632247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.632168 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx"] Nov 02 17:52:52.636544 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.636522 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.639736 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.639712 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.639869 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.639760 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.639869 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.639789 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.639869 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.639819 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gn2\" (UniqueName: \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-kube-api-access-79gn2\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.640030 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.639912 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv5k2\" (UniqueName: \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-kube-api-access-vv5k2\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.640152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640129 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.640212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640193 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10d03bf3-ecba-4349-ac6c-0410e573d494-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.640292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640220 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.640292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640265 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.640400 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640296 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.640400 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640348 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.640400 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640378 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.640564 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640411 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.640564 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640454 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.640564 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640472 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.640564 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640492 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.640564 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640514 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.640928 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640908 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10d03bf3-ecba-4349-ac6c-0410e573d494-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.641000 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.640967 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.641143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.641122 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.641231 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.641185 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.641479 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.641456 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.641584 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.641520 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.642693 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.642672 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.643393 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.643373 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.665451 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.665406 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx"] Nov 02 17:52:52.669311 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.669283 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.677194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.677110 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gn2\" (UniqueName: \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-kube-api-access-79gn2\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.742821 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.742780 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.743007 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.742838 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.743007 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.742968 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.743132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743024 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbflh\" (UniqueName: \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-kube-api-access-rbflh\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.743174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743152 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.743213 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743180 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.743249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743230 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv5k2\" (UniqueName: \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-kube-api-access-vv5k2\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.743320 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743305 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.743375 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743360 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.743426 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743390 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.743517 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743478 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.743517 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743502 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.743633 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743552 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-data\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.743633 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743569 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.743739 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743631 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.743739 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743694 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.743739 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743723 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.743881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743767 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.743934 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743891 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.743994 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743931 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.743994 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.743980 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-certs\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.744138 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.744106 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-credential-socket\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.744517 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.744492 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istiod-ca-cert\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.745387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.745360 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-envoy\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.746170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.746148 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-podinfo\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.752579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.752554 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-token\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.753773 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.753754 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv5k2\" (UniqueName: \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-kube-api-access-vv5k2\") pod \"e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.761612 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.761582 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:52.779766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.779740 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:52.857454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.857214 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.857454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.857278 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.857454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.857328 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.857454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.857396 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.857898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.857507 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.857968 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.857925 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.858405 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.858185 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.858405 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.858210 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbflh\" (UniqueName: \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-kube-api-access-rbflh\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.858548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.858405 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.858548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.858414 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.858548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.858489 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.858548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.858519 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.859809 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.859036 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.860531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.860444 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.862978 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.862954 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.863280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.863265 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.867178 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.867155 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.869246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.869224 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbflh\" (UniqueName: \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-kube-api-access-rbflh\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:52.988367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:52.988334 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:53.060761 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:53.060707 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d03bf3_ecba_4349_ac6c_0410e573d494.slice/crio-0195302e3687a114d04c00c0b91f4946fd43986c0c760dccd4ca2ce84b974ba5 WatchSource:0}: Error finding container 0195302e3687a114d04c00c0b91f4946fd43986c0c760dccd4ca2ce84b974ba5: Status 404 returned error can't find the container with id 0195302e3687a114d04c00c0b91f4946fd43986c0c760dccd4ca2ce84b974ba5 Nov 02 17:52:53.062426 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.061965 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2"] Nov 02 17:52:53.064899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.064344 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:53.064899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.064420 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:53.064899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.064458 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:53.093940 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.093873 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs"] Nov 02 17:52:53.098616 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:53.098582 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0558ce75_6fbc_442b_8024_6b66f00e5f9f.slice/crio-7aa6a9651464a89bc190e4aa732a0ee840f906c00ffefadfc39a1a1dc3e0860d WatchSource:0}: Error finding container 7aa6a9651464a89bc190e4aa732a0ee840f906c00ffefadfc39a1a1dc3e0860d: Status 404 returned error can't find the container with id 7aa6a9651464a89bc190e4aa732a0ee840f906c00ffefadfc39a1a1dc3e0860d Nov 02 17:52:53.102046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.102014 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:53.102159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.102116 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:53.102209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.102160 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:53.303095 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.302497 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx"] Nov 02 17:52:53.303658 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:52:53.303620 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2997e8d_e663_4e09_b1ea_a2a796d00aaa.slice/crio-c801ba8ddd676d7987deb631f88ceec35661d8f7bb5a11a033ae70eff8e051af WatchSource:0}: Error finding container c801ba8ddd676d7987deb631f88ceec35661d8f7bb5a11a033ae70eff8e051af: Status 404 returned error can't find the container with id c801ba8ddd676d7987deb631f88ceec35661d8f7bb5a11a033ae70eff8e051af Nov 02 17:52:53.306574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.306538 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:53.306687 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.306669 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:53.306747 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.306731 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:52:53.321231 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.321202 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:53.701192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.691119 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs"] Nov 02 17:52:53.701192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.696994 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:53.713430 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.713399 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:52:53.714377 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.714352 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:52:53.727941 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.727908 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs"] Nov 02 17:52:53.798146 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.798064 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0c60ca9-nwrfs\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:53.798347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.798176 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0c60ca9-nwrfs\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:53.798989 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.798945 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wn57\" (UniqueName: \"kubernetes.io/projected/051ab575-90bd-46d9-9123-9ba74707d4b7-kube-api-access-2wn57\") pod \"maas-api-revoke-sub-a0c60ca9-nwrfs\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:53.908101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.908042 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wn57\" (UniqueName: \"kubernetes.io/projected/051ab575-90bd-46d9-9123-9ba74707d4b7-kube-api-access-2wn57\") pod \"maas-api-revoke-sub-a0c60ca9-nwrfs\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:53.908288 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.908225 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0c60ca9-nwrfs\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:53.908339 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.908298 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0c60ca9-nwrfs\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:53.909194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.909166 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0c60ca9-nwrfs\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:53.910977 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.910945 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0c60ca9-nwrfs\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:53.932235 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.932197 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wn57\" (UniqueName: \"kubernetes.io/projected/051ab575-90bd-46d9-9123-9ba74707d4b7-kube-api-access-2wn57\") pod \"maas-api-revoke-sub-a0c60ca9-nwrfs\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:53.967191 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.955029 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" event={"ID":"10d03bf3-ecba-4349-ac6c-0410e573d494","Type":"ContainerStarted","Data":"adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea"} Nov 02 17:52:53.995407 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.995376 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:52:53.995686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.995665 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" event={"ID":"10d03bf3-ecba-4349-ac6c-0410e573d494","Type":"ContainerStarted","Data":"0195302e3687a114d04c00c0b91f4946fd43986c0c760dccd4ca2ce84b974ba5"} Nov 02 17:52:53.995826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.995811 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" event={"ID":"b2997e8d-e663-4e09-b1ea-a2a796d00aaa","Type":"ContainerStarted","Data":"3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5"} Nov 02 17:52:53.995926 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.995914 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" event={"ID":"b2997e8d-e663-4e09-b1ea-a2a796d00aaa","Type":"ContainerStarted","Data":"c801ba8ddd676d7987deb631f88ceec35661d8f7bb5a11a033ae70eff8e051af"} Nov 02 17:52:53.996049 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.996036 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" event={"ID":"0558ce75-6fbc-442b-8024-6b66f00e5f9f","Type":"ContainerStarted","Data":"036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f"} Nov 02 17:52:53.996190 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:53.996165 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" event={"ID":"0558ce75-6fbc-442b-8024-6b66f00e5f9f","Type":"ContainerStarted","Data":"7aa6a9651464a89bc190e4aa732a0ee840f906c00ffefadfc39a1a1dc3e0860d"} Nov 02 17:52:54.071101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.070318 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:54.094145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.088369 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podStartSLOduration=2.088345125 podStartE2EDuration="2.088345125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:54.042286056 +0000 UTC m=+771.485368694" watchObservedRunningTime="2026-11-02 17:52:54.088345125 +0000 UTC m=+771.531427763" Nov 02 17:52:54.104953 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.104878 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podStartSLOduration=2.104859863 podStartE2EDuration="2.104859863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:54.104479549 +0000 UTC m=+771.547562196" watchObservedRunningTime="2026-11-02 17:52:54.104859863 +0000 UTC m=+771.547942502" Nov 02 17:52:54.172106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.168195 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podStartSLOduration=2.168170287 podStartE2EDuration="2.168170287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:54.140409419 +0000 UTC m=+771.583492075" watchObservedRunningTime="2026-11-02 17:52:54.168170287 +0000 UTC m=+771.611252923" Nov 02 17:52:54.534161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.534030 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs"] Nov 02 17:52:54.795157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.794980 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:52:54.796115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.795370 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:52:54.797303 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.797232 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:52:54.797473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.797273 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:52:54.801114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.798195 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:52:54.801114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.798247 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:52:54.974607 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.974471 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:54.994378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.993983 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 17:52:54.994378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:54.994051 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 17:52:55.037101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:55.035381 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" event={"ID":"051ab575-90bd-46d9-9123-9ba74707d4b7","Type":"ContainerStarted","Data":"23267d8fe09e56b667be47ae06c2ae0f7820b2445b37553a25ff19203ced9b8f"} Nov 02 17:52:55.861130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:55.857392 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:52:55.861130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:55.857476 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:52:55.861130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:55.857653 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:52:55.861130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:55.857715 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:52:55.861130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:55.858834 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:56.010984 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:55.993482 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 17:52:56.010984 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:55.993540 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 17:52:56.232756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:56.232721 4310 generic.go:356] "Generic (PLEG): container finished" podID="051ab575-90bd-46d9-9123-9ba74707d4b7" containerID="9b09e444a68b50f6a49a876f28a33e248c61bc508d7b487d522bb9085d5c5464" exitCode=0 Nov 02 17:52:56.233473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:56.233148 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" event={"ID":"051ab575-90bd-46d9-9123-9ba74707d4b7","Type":"ContainerDied","Data":"9b09e444a68b50f6a49a876f28a33e248c61bc508d7b487d522bb9085d5c5464"} Nov 02 17:52:56.766103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:56.754836 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:56.768026 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:56.767985 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:52:56.768201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:56.768053 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:52:56.800444 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:56.800408 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:52:56.800721 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:56.800686 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:52:57.014102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.003135 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 17:52:57.014102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.003206 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 17:52:57.808133 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.807506 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:52:57.808133 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.807567 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:52:57.808133 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.807939 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:52:57.808133 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.807973 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:52:57.837205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.837150 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:57.944283 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.933734 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:57.945680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.945646 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-xks-ca\") pod \"051ab575-90bd-46d9-9123-9ba74707d4b7\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " Nov 02 17:52:57.945924 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.945903 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051ab575-90bd-46d9-9123-9ba74707d4b7-kube-api-access-2wn57\" (UniqueName: \"kubernetes.io/projected/051ab575-90bd-46d9-9123-9ba74707d4b7-kube-api-access-2wn57\") pod \"051ab575-90bd-46d9-9123-9ba74707d4b7\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " Nov 02 17:52:57.946157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.946134 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-openshift-ca\") pod \"051ab575-90bd-46d9-9123-9ba74707d4b7\" (UID: \"051ab575-90bd-46d9-9123-9ba74707d4b7\") " Nov 02 17:52:57.958338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:57.958297 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-openshift-ca" pod "051ab575-90bd-46d9-9123-9ba74707d4b7" (UID: "051ab575-90bd-46d9-9123-9ba74707d4b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:58.034140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.010818 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 17:52:58.034140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.010895 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 17:52:58.085101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.065692 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.240158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.239811 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-xks-ca" pod "051ab575-90bd-46d9-9123-9ba74707d4b7" (UID: "051ab575-90bd-46d9-9123-9ba74707d4b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:58.241004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.240963 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051ab575-90bd-46d9-9123-9ba74707d4b7-kube-api-access-2wn57" pod "051ab575-90bd-46d9-9123-9ba74707d4b7" (UID: "051ab575-90bd-46d9-9123-9ba74707d4b7"). InnerVolumeSpecName "kube-api-access-2wn57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:58.331136 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.331065 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ab575-90bd-46d9-9123-9ba74707d4b7-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.331136 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.331139 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wn57\" (UniqueName: \"kubernetes.io/projected/051ab575-90bd-46d9-9123-9ba74707d4b7-kube-api-access-2wn57\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:52:58.335762 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.335729 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" event={"ID":"051ab575-90bd-46d9-9123-9ba74707d4b7","Type":"ContainerDied","Data":"23267d8fe09e56b667be47ae06c2ae0f7820b2445b37553a25ff19203ced9b8f"} Nov 02 17:52:58.335893 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.335786 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23267d8fe09e56b667be47ae06c2ae0f7820b2445b37553a25ff19203ced9b8f" Nov 02 17:52:58.335993 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.335749 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs" Nov 02 17:52:58.762966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.762485 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:52:58.762966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.762571 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:52:58.797697 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.797656 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:52:58.798009 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:58.797980 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:52:59.009576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:59.009540 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 17:52:59.009847 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:59.009815 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 17:52:59.041441 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:59.041337 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:52:59.763623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:59.763588 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:52:59.763945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:59.763893 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:52:59.782709 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:59.782675 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:52:59.782940 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:59.782914 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:52:59.990637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:59.990595 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 17:52:59.990921 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:52:59.990894 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 17:53:00.251135 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:00.241922 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:00.762114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:00.762056 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 17:53:00.762320 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:00.762143 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 17:53:00.780822 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:00.780781 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:53:00.780973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:00.780867 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 17:53:00.988863 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:00.988822 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 17:53:00.989049 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:00.988903 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 17:53:01.170320 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:01.170189 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:01.800924 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:01.800888 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:53:01.803148 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:01.803123 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:53:01.805556 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:01.805534 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 17:53:01.900247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:01.899370 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:53:01.912435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:01.909155 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:53:01.931119 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:01.927732 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 17:53:02.174149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:02.173999 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:53:02.174657 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:02.174632 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:53:02.175684 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:02.175664 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:53:02.263616 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:02.263579 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:03.524736 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:53:03.524698 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:53:03.554068 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:03.554020 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:03.963586 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:03.963560 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv"] Nov 02 17:53:03.964240 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:03.964226 4310 cpu_manager.go:414] "containerMap: removing container" podUID="051ab575-90bd-46d9-9123-9ba74707d4b7" containerName="revoke-keys" Nov 02 17:53:03.964307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:03.964242 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051ab575-90bd-46d9-9123-9ba74707d4b7" containerName="revoke-keys" Nov 02 17:53:03.964358 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:03.964348 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="051ab575-90bd-46d9-9123-9ba74707d4b7" containerName="revoke-keys" Nov 02 17:53:03.969386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:03.969365 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:03.981456 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:03.981427 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv"] Nov 02 17:53:03.982537 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:03.982512 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:53:03.982665 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:03.982613 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:53:04.056984 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.056938 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsl52\" (UniqueName: \"kubernetes.io/projected/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-kube-api-access-gsl52\") pod \"maas-api-revoke-sub-611811fe-8bhkv\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:04.057208 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.057185 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-611811fe-8bhkv\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:04.057280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.057254 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-611811fe-8bhkv\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:04.159137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.159057 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-611811fe-8bhkv\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:04.159339 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.159278 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsl52\" (UniqueName: \"kubernetes.io/projected/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-kube-api-access-gsl52\") pod \"maas-api-revoke-sub-611811fe-8bhkv\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:04.159383 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.159375 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-611811fe-8bhkv\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:04.160049 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.160022 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-611811fe-8bhkv\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:04.161510 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.161485 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-611811fe-8bhkv\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:04.172489 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.172460 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsl52\" (UniqueName: \"kubernetes.io/projected/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-kube-api-access-gsl52\") pod \"maas-api-revoke-sub-611811fe-8bhkv\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:04.290696 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.290598 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:04.380399 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.380349 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:04.723156 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:04.723119 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc92f38_7bd6_45b6_999b_5bb7a84d1ec6.slice/crio-69fc8382b701a02b12254007fdf810bf3710541da71f60be392a896ff986bc63 WatchSource:0}: Error finding container 69fc8382b701a02b12254007fdf810bf3710541da71f60be392a896ff986bc63: Status 404 returned error can't find the container with id 69fc8382b701a02b12254007fdf810bf3710541da71f60be392a896ff986bc63 Nov 02 17:53:04.723552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:04.723402 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv"] Nov 02 17:53:05.185353 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.185320 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw"] Nov 02 17:53:05.189605 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.189577 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.203882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.203852 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"default-dockercfg-jptzs\"" Nov 02 17:53:05.204018 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.203857 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"distinct-2-w6-7b0683-kserve-self-signed-certs\"" Nov 02 17:53:05.207574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.207545 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d27280-8d99-4708-be3a-e4781a0c43b0-tls-certs\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.207766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.207742 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-model-cache\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.207888 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.207867 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-home\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.208019 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.207971 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzccq\" (UniqueName: \"kubernetes.io/projected/29d27280-8d99-4708-be3a-e4781a0c43b0-kube-api-access-wzccq\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.208174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.208157 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-dshm\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.208277 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.208249 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-tmp-dir\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.212024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.212000 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"openshift-service-ca.crt\"" Nov 02 17:53:05.212208 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.212072 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7b0683\"/\"kube-root-ca.crt\"" Nov 02 17:53:05.213828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.213802 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw"] Nov 02 17:53:05.252858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.252804 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:05.338165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.313304 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzccq\" (UniqueName: \"kubernetes.io/projected/29d27280-8d99-4708-be3a-e4781a0c43b0-kube-api-access-wzccq\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.338165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.313536 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-dshm\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.338165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.313578 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-tmp-dir\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.338165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.313706 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d27280-8d99-4708-be3a-e4781a0c43b0-tls-certs\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.338165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.313746 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-model-cache\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.338165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.313876 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-home\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.338165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.314499 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-home\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.352101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.350820 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d27280-8d99-4708-be3a-e4781a0c43b0-tls-certs\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.352101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.351155 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-tmp-dir\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.352101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.351409 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-model-cache\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.366414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.366345 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-dshm\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.382102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.381600 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzccq\" (UniqueName: \"kubernetes.io/projected/29d27280-8d99-4708-be3a-e4781a0c43b0-kube-api-access-wzccq\") pod \"distinct-2-w6-7b0683-kserve-7866445f8-ntfdw\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.523593 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.521356 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:05.547051 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.547004 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" event={"ID":"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6","Type":"ContainerStarted","Data":"b836eac27b9f00fcc13bc6fb659fbda1bca4b11b918102dad54d20c9634838ea"} Nov 02 17:53:05.547281 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.547148 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" event={"ID":"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6","Type":"ContainerStarted","Data":"69fc8382b701a02b12254007fdf810bf3710541da71f60be392a896ff986bc63"} Nov 02 17:53:05.572909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.572590 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" podStartSLOduration=2.572572315 podStartE2EDuration="2.572572315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:05.569535294 +0000 UTC m=+783.012617927" watchObservedRunningTime="2026-11-02 17:53:05.572572315 +0000 UTC m=+783.015654950" Nov 02 17:53:05.672487 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.672369 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:05.950827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:05.950733 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw"] Nov 02 17:53:05.963881 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:05.963858 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29d27280_8d99_4708_be3a_e4781a0c43b0.slice/crio-a1aed7690944c5662e89408582f7328c01b6434c54787356a6d34f5e0581bca6 WatchSource:0}: Error finding container a1aed7690944c5662e89408582f7328c01b6434c54787356a6d34f5e0581bca6: Status 404 returned error can't find the container with id a1aed7690944c5662e89408582f7328c01b6434c54787356a6d34f5e0581bca6 Nov 02 17:53:06.558394 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:06.558360 4310 generic.go:356] "Generic (PLEG): container finished" podID="4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6" containerID="b836eac27b9f00fcc13bc6fb659fbda1bca4b11b918102dad54d20c9634838ea" exitCode=0 Nov 02 17:53:06.558595 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:06.558438 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" event={"ID":"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6","Type":"ContainerDied","Data":"b836eac27b9f00fcc13bc6fb659fbda1bca4b11b918102dad54d20c9634838ea"} Nov 02 17:53:06.560478 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:06.560448 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" event={"ID":"29d27280-8d99-4708-be3a-e4781a0c43b0","Type":"ContainerStarted","Data":"cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a"} Nov 02 17:53:06.560626 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:06.560489 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" event={"ID":"29d27280-8d99-4708-be3a-e4781a0c43b0","Type":"ContainerStarted","Data":"a1aed7690944c5662e89408582f7328c01b6434c54787356a6d34f5e0581bca6"} Nov 02 17:53:06.627019 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:06.626952 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podStartSLOduration=1.626932909 podStartE2EDuration="1.626932909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:06.624752623 +0000 UTC m=+784.067835260" watchObservedRunningTime="2026-11-02 17:53:06.626932909 +0000 UTC m=+784.070015547" Nov 02 17:53:06.919249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:06.919159 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:07.250522 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.250492 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj"] Nov 02 17:53:07.509970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.503346 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj"] Nov 02 17:53:07.509970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.503483 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.530109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.528405 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:07.557102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.552462 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.593096 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.566320 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.593096 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.566405 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.593096 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.566595 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7ckk\" (UniqueName: \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-kube-api-access-v7ckk\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.593096 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.566643 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5d24c0-427d-4b1c-b706-965be02d81d0-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.593096 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.566696 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.593096 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.566739 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.593096 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.566846 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.593096 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.566894 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.737112 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.737183 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.737296 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.737347 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.737393 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.737463 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.737517 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.737665 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ckk\" (UniqueName: \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-kube-api-access-v7ckk\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.737723 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5d24c0-427d-4b1c-b706-965be02d81d0-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.738763 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5d24c0-427d-4b1c-b706-965be02d81d0-istiod-ca-cert\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.739137 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-credential-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.773156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.739449 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-socket\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.809101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.775839 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-certs\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.809101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.776131 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-data\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.955966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.955576 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7ckk\" (UniqueName: \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-kube-api-access-v7ckk\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.961478 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.960653 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-token\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.961478 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.960918 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-envoy\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.997048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.976287 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-podinfo\") pod \"e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:07.997048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:07.994422 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:08.046491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.036461 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:08.205676 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.204572 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:08.271101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.267400 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w"] Nov 02 17:53:08.301449 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.301363 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.345104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.324150 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.367262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.367225 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.388110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.371274 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.388598 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.388568 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbq9v\" (UniqueName: \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-kube-api-access-zbq9v\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.394443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.394409 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.408219 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.408186 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d15b4f-697d-40ac-9299-fbf4b413127c-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.408505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.408485 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.408672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.408656 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.482186 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.482155 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.529327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.527246 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74"] Nov 02 17:53:08.549967 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.549936 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.823366 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xk2n\" (UniqueName: \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-kube-api-access-4xk2n\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.823452 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.823507 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.823547 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d15b4f-697d-40ac-9299-fbf4b413127c-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.823595 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.823627 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.823691 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.823771 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.823840 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.823883 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.823918 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.824022 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.824104 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.824159 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.824271 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.824326 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.824377 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.824433 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbq9v\" (UniqueName: \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-kube-api-access-zbq9v\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.825478 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.826357 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d15b4f-697d-40ac-9299-fbf4b413127c-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.831177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.826746 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.862647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.862546 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.873624 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.873340 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.913101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.908451 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.913101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.908572 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:08.913101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.910070 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:08.937040 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.936995 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.942225 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.942185 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.980245 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.980207 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:08.986243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.976027 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:53:09.005117 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.998549 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:09.005117 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.998722 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xk2n\" (UniqueName: \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-kube-api-access-4xk2n\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:09.005117 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.998795 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:09.005117 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.998915 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:09.005117 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.999048 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:09.005117 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.999195 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:09.005117 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:08.999250 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:09.055601 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.049396 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-data\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:09.055601 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.049521 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:09.059099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.056485 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istiod-ca-cert\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:09.664847 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.664808 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbq9v\" (UniqueName: \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-kube-api-access-zbq9v\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:09.666650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.666609 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:09.701003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.667422 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-credential-socket\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:09.701289 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.667893 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:09.701499 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.667940 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded" Nov 02 17:53:09.825274 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.825214 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:09.870889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.870807 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-certs\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:09.928362 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.928245 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:09.973103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:09.966532 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:10.020099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.017301 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:53:10.073599 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.073533 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:10.611521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.446053 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:53:10.611521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.448053 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:10.611521 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:53:10.491358 4310 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.8s" Nov 02 17:53:10.611521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.491400 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w"] Nov 02 17:53:10.611521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.491496 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:53:10.611521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.491534 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:53:10.611521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.491569 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:53:10.611521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.491604 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:53:10.611521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.501324 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:10.611521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.515592 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74"] Nov 02 17:53:10.696104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.692730 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-envoy\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:10.714111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.701288 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-698c564c8b-v5qgk" podUID="d46ea576-9209-41ed-96c0-9ced3e50d3ae" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:10.828100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.820145 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded" Nov 02 17:53:10.828100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.820268 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" podUID="3b41c5ba-f619-4894-8e4f-427014637518" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Nov 02 17:53:10.850728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:10.848015 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:53:11.325103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.274154 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:11.325103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.294489 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" podUID="c19cdb66-3f5e-4ac8-a77c-d0a85d130613" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Nov 02 17:53:11.408545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.399047 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-podinfo\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:11.408545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.403703 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:11.513443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.448768 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:11.513443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.458031 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:11.564106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.545784 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-token\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:11.593062 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.592967 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:11.669580 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.644491 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xk2n\" (UniqueName: \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-kube-api-access-4xk2n\") pod \"e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:11.722194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.720536 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:11.785309 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.741331 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:11.785670 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.785629 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:11.785862 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.785832 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:11.833229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.833186 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:53:11.833501 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.833482 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:53:11.834138 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.834119 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:53:11.846143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.845181 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:53:11.846143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.845231 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:53:11.846143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.845265 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:53:11.846143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.845296 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:53:11.846143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.845324 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:53:11.846143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.835060 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:11.866118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.859707 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj"] Nov 02 17:53:11.990371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.978241 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-xks-ca\") pod \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " Nov 02 17:53:11.990371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.978306 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-kube-api-access-gsl52\" (UniqueName: \"kubernetes.io/projected/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-kube-api-access-gsl52\") pod \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " Nov 02 17:53:11.990371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.978387 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-openshift-ca\") pod \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\" (UID: \"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6\") " Nov 02 17:53:11.995868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.995815 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-openshift-ca" pod "4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6" (UID: "4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:11.996429 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:11.996402 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:12.002756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.002730 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-xks-ca" pod "4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6" (UID: "4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:12.006675 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.006634 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-kube-api-access-gsl52" pod "4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6" (UID: "4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6"). InnerVolumeSpecName "kube-api-access-gsl52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:12.097699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.097625 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:12.097866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.097853 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsl52\" (UniqueName: \"kubernetes.io/projected/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6-kube-api-access-gsl52\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:12.189397 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.189162 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:12.189397 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.189263 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:12.189397 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.189313 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:12.196654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.196580 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:12.201364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.201287 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w"] Nov 02 17:53:12.207012 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.206979 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74"] Nov 02 17:53:12.215204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.215124 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:12.215204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.215196 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:12.215375 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.215249 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:12.427931 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.427902 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" event={"ID":"ca5d24c0-427d-4b1c-b706-965be02d81d0","Type":"ContainerStarted","Data":"87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf"} Nov 02 17:53:12.428253 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.428216 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" event={"ID":"ca5d24c0-427d-4b1c-b706-965be02d81d0","Type":"ContainerStarted","Data":"456a10cac7878eab7e887c5587c3bf69068cb34804b4585fbb8ba595089731c4"} Nov 02 17:53:12.433000 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.432984 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" Nov 02 17:53:12.434025 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.433993 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv" event={"ID":"4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6","Type":"ContainerDied","Data":"69fc8382b701a02b12254007fdf810bf3710541da71f60be392a896ff986bc63"} Nov 02 17:53:12.434112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.434057 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69fc8382b701a02b12254007fdf810bf3710541da71f60be392a896ff986bc63" Nov 02 17:53:12.439895 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.439168 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" event={"ID":"6069e34a-83a9-4812-85d4-ef0fd0e5173d","Type":"ContainerStarted","Data":"5783789197b4eac816f8e1ef90e8d2d94cbc8b1706c57ff5fbc86840c84dc857"} Nov 02 17:53:12.443955 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.442578 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" event={"ID":"38d15b4f-697d-40ac-9299-fbf4b413127c","Type":"ContainerStarted","Data":"0a5e1d6932d7c4e61fde4b2bd1a2291e6ff6fd72b221eece6a8ce60cb1811854"} Nov 02 17:53:12.478174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.478062 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" podStartSLOduration=5.478042806 podStartE2EDuration="5.478042806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:12.474398908 +0000 UTC m=+789.917481545" watchObservedRunningTime="2026-11-02 17:53:12.478042806 +0000 UTC m=+789.921125443" Nov 02 17:53:12.513365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.513310 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" podStartSLOduration=5.513295674 podStartE2EDuration="5.513295674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:12.506333979 +0000 UTC m=+789.949416616" watchObservedRunningTime="2026-11-02 17:53:12.513295674 +0000 UTC m=+789.956378309" Nov 02 17:53:12.542544 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.542487 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" podStartSLOduration=5.5424712320000005 podStartE2EDuration="5.542471232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:12.538510569 +0000 UTC m=+789.981593205" watchObservedRunningTime="2026-11-02 17:53:12.542471232 +0000 UTC m=+789.985553859" Nov 02 17:53:12.720764 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:12.720724 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:13.219187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:13.219142 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:13.222258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:13.219517 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:53:13.222635 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:13.222501 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 17:53:13.288445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:13.288399 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:13.307101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:13.300253 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:53:13.307101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:13.300336 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:53:13.351776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:13.351739 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:13.502200 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:13.501251 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" event={"ID":"6069e34a-83a9-4812-85d4-ef0fd0e5173d","Type":"ContainerStarted","Data":"40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0"} Nov 02 17:53:13.524104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:13.518522 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" event={"ID":"38d15b4f-697d-40ac-9299-fbf4b413127c","Type":"ContainerStarted","Data":"2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53"} Nov 02 17:53:13.741944 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:13.741223 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:53:13.741944 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:13.741286 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:53:14.219102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:14.218786 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:53:14.219102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:14.218859 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 17:53:14.279109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:14.278506 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:53:14.279109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:14.278574 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:53:14.735108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:14.733896 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:14.760222 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:14.741381 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:53:14.760222 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:14.741455 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:53:15.224807 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:15.224680 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:53:15.225557 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:15.225428 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 17:53:15.289274 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:15.289226 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:53:15.289569 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:15.289525 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:53:15.523651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:15.523565 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:15.538486 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:15.538457 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:15.642114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:15.639327 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:15.918284 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:15.918170 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:53:15.918576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:15.918543 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:53:15.975528 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:15.975484 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:16.305109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:16.300134 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:53:16.305109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:16.300223 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:53:16.352636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:16.352581 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:53:16.361240 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:16.361190 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 17:53:16.719163 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:16.713602 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:16.734354 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:16.733298 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:53:16.734354 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:16.733363 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:53:16.832300 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:16.831238 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:16.853403 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:16.853359 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:17.187888 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:17.187778 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:17.222022 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:17.221510 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:53:17.222022 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:17.221599 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 17:53:17.277688 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:17.277644 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:53:17.278001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:17.277955 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:53:17.759268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:17.737445 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:53:17.759268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:17.737516 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:53:17.956652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:17.939149 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf"] Nov 02 17:53:17.956652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:17.940028 4310 cpu_manager.go:414] "containerMap: removing container" podUID="4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6" containerName="revoke-keys" Nov 02 17:53:17.956652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:17.940050 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6" containerName="revoke-keys" Nov 02 17:53:17.956652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:17.940227 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6" containerName="revoke-keys" Nov 02 17:53:18.032292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.031036 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:18.080898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.080857 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17bc4191-zvwkf\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:18.081242 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.081213 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17bc4191-zvwkf\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:18.091273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.081717 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:53:18.091273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.082418 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:53:18.091807 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.091776 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n4kd\" (UniqueName: \"kubernetes.io/projected/c8519447-24ff-4cb9-8856-47bb7fa04b60-kube-api-access-6n4kd\") pod \"maas-api-revoke-sub-17bc4191-zvwkf\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:18.241630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.187636 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf"] Nov 02 17:53:18.267102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.197613 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17bc4191-zvwkf\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:18.267102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.247483 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n4kd\" (UniqueName: \"kubernetes.io/projected/c8519447-24ff-4cb9-8856-47bb7fa04b60-kube-api-access-6n4kd\") pod \"maas-api-revoke-sub-17bc4191-zvwkf\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:18.267102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.247747 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17bc4191-zvwkf\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:18.267102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.248716 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17bc4191-zvwkf\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:18.267102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.217545 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:53:18.267102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.248795 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 17:53:18.326196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.306637 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:53:18.326196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.306702 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:53:18.364679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.364633 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17bc4191-zvwkf\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:18.394896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.388572 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n4kd\" (UniqueName: \"kubernetes.io/projected/c8519447-24ff-4cb9-8856-47bb7fa04b60-kube-api-access-6n4kd\") pod \"maas-api-revoke-sub-17bc4191-zvwkf\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:18.622565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.622472 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:18.738875 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.738839 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:53:18.739176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.739143 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:53:18.741318 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:18.741289 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:19.220295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:19.220257 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:53:19.220842 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:19.220811 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 17:53:19.323287 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:19.323241 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:53:19.323561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:19.323527 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:53:19.585969 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:19.585932 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf"] Nov 02 17:53:19.721574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:19.721534 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:53:19.721725 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:19.721618 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 17:53:19.878593 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:19.878543 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" event={"ID":"c8519447-24ff-4cb9-8856-47bb7fa04b60","Type":"ContainerStarted","Data":"278c352b787f4caa3ea1a33d8a7ef92011435561946bca2e8e11d462cb755df6"} Nov 02 17:53:19.911904 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:19.911846 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:20.218295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:20.218241 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:53:20.218551 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:20.218310 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 17:53:20.275712 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:20.275672 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 17:53:20.276139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:20.275744 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 17:53:20.904730 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:20.904682 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" event={"ID":"c8519447-24ff-4cb9-8856-47bb7fa04b60","Type":"ContainerStarted","Data":"9ee9e88dcae8e17ad8b849535462abd74df7babfb243349b1abeda848fa8f2f2"} Nov 02 17:53:20.931468 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:20.931433 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:20.933499 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:20.933460 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:20.939896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:20.939865 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:53:21.098565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.098478 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:21.251452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.251422 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:21.253842 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.253775 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:21.258657 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.258629 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 17:53:21.301149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.301108 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:21.315323 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.315293 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:21.343290 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.343260 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:53:21.357211 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.354940 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv"] Nov 02 17:53:21.370210 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.370182 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv"] Nov 02 17:53:21.370539 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.370518 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:21.405797 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.405157 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a15ff56-jjvdv\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:21.406045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.406018 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2vcm\" (UniqueName: \"kubernetes.io/projected/15a19313-0559-4a24-956b-2b52cb2769bb-kube-api-access-s2vcm\") pod \"maas-api-revoke-sub-3a15ff56-jjvdv\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:21.406383 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.406365 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a15ff56-jjvdv\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:21.510208 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.510165 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a15ff56-jjvdv\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:21.511045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.511018 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2vcm\" (UniqueName: \"kubernetes.io/projected/15a19313-0559-4a24-956b-2b52cb2769bb-kube-api-access-s2vcm\") pod \"maas-api-revoke-sub-3a15ff56-jjvdv\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:21.511861 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.511823 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a15ff56-jjvdv\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:21.513066 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.513040 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a15ff56-jjvdv\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:21.513876 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.513856 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a15ff56-jjvdv\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:21.521261 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.521235 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2vcm\" (UniqueName: \"kubernetes.io/projected/15a19313-0559-4a24-956b-2b52cb2769bb-kube-api-access-s2vcm\") pod \"maas-api-revoke-sub-3a15ff56-jjvdv\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:21.704138 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.702144 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:21.915818 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.915782 4310 generic.go:356] "Generic (PLEG): container finished" podID="c8519447-24ff-4cb9-8856-47bb7fa04b60" containerID="9ee9e88dcae8e17ad8b849535462abd74df7babfb243349b1abeda848fa8f2f2" exitCode=0 Nov 02 17:53:21.915982 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:21.915865 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" event={"ID":"c8519447-24ff-4cb9-8856-47bb7fa04b60","Type":"ContainerDied","Data":"9ee9e88dcae8e17ad8b849535462abd74df7babfb243349b1abeda848fa8f2f2"} Nov 02 17:53:22.004693 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.004607 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv"] Nov 02 17:53:22.006880 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:22.006855 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a19313_0559_4a24_956b_2b52cb2769bb.slice/crio-85c0da5fe0e942f202f124d03854a1b9c6300647ab66b50570b2a89b1de21587 WatchSource:0}: Error finding container 85c0da5fe0e942f202f124d03854a1b9c6300647ab66b50570b2a89b1de21587: Status 404 returned error can't find the container with id 85c0da5fe0e942f202f124d03854a1b9c6300647ab66b50570b2a89b1de21587 Nov 02 17:53:22.126732 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.126703 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:22.207291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.207253 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:22.241461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.241270 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-xks-ca\") pod \"c8519447-24ff-4cb9-8856-47bb7fa04b60\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " Nov 02 17:53:22.242803 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.242746 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-openshift-ca" pod "c8519447-24ff-4cb9-8856-47bb7fa04b60" (UID: "c8519447-24ff-4cb9-8856-47bb7fa04b60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:22.243062 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.241995 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-openshift-ca\") pod \"c8519447-24ff-4cb9-8856-47bb7fa04b60\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " Nov 02 17:53:22.263413 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.263381 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8519447-24ff-4cb9-8856-47bb7fa04b60-kube-api-access-6n4kd\" (UniqueName: \"kubernetes.io/projected/c8519447-24ff-4cb9-8856-47bb7fa04b60-kube-api-access-6n4kd\") pod \"c8519447-24ff-4cb9-8856-47bb7fa04b60\" (UID: \"c8519447-24ff-4cb9-8856-47bb7fa04b60\") " Nov 02 17:53:22.270115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.264566 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:22.288097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.278195 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-xks-ca" pod "c8519447-24ff-4cb9-8856-47bb7fa04b60" (UID: "c8519447-24ff-4cb9-8856-47bb7fa04b60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:22.298099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.293119 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8519447-24ff-4cb9-8856-47bb7fa04b60-kube-api-access-6n4kd" pod "c8519447-24ff-4cb9-8856-47bb7fa04b60" (UID: "c8519447-24ff-4cb9-8856-47bb7fa04b60"). InnerVolumeSpecName "kube-api-access-6n4kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:22.383970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.383932 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n4kd\" (UniqueName: \"kubernetes.io/projected/c8519447-24ff-4cb9-8856-47bb7fa04b60-kube-api-access-6n4kd\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:22.383970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.383974 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8519447-24ff-4cb9-8856-47bb7fa04b60-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:22.389325 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.389145 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk"] Nov 02 17:53:22.392350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.390223 4310 cpu_manager.go:414] "containerMap: removing container" podUID="c8519447-24ff-4cb9-8856-47bb7fa04b60" containerName="revoke-keys" Nov 02 17:53:22.392350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.390245 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8519447-24ff-4cb9-8856-47bb7fa04b60" containerName="revoke-keys" Nov 02 17:53:22.392350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.390447 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8519447-24ff-4cb9-8856-47bb7fa04b60" containerName="revoke-keys" Nov 02 17:53:22.399745 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.396829 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.399745 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.397947 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk"] Nov 02 17:53:22.484898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.484858 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557797b3-5dcf-4373-8a05-a8aa9fba451e-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.485051 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.484908 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.485051 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.484999 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghsh\" (UniqueName: \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-kube-api-access-2ghsh\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.485051 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.485041 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.485187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.485070 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.485187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.485132 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.485329 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.485307 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.485438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.485426 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.485495 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.485485 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.586741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.586711 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghsh\" (UniqueName: \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-kube-api-access-2ghsh\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.586905 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.586753 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.586905 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.586778 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.586905 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.586823 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.587061 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.586997 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.587170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.587149 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.587242 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.587197 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.587293 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.587232 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.587381 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.587362 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557797b3-5dcf-4373-8a05-a8aa9fba451e-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.587461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.587433 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.587520 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.587509 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.587573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.587554 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.587832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.587783 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.588147 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.588130 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557797b3-5dcf-4373-8a05-a8aa9fba451e-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.589258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.589240 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.591861 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.591833 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.596583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.596561 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.597027 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.597004 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghsh\" (UniqueName: \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-kube-api-access-2ghsh\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.726727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.726691 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:22.803278 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.803201 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p"] Nov 02 17:53:22.850141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.846814 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p"] Nov 02 17:53:22.850141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.846933 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:22.926900 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.926873 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" Nov 02 17:53:22.927189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.926900 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf" event={"ID":"c8519447-24ff-4cb9-8856-47bb7fa04b60","Type":"ContainerDied","Data":"278c352b787f4caa3ea1a33d8a7ef92011435561946bca2e8e11d462cb755df6"} Nov 02 17:53:22.927189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.926946 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="278c352b787f4caa3ea1a33d8a7ef92011435561946bca2e8e11d462cb755df6" Nov 02 17:53:22.928959 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.928934 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" event={"ID":"15a19313-0559-4a24-956b-2b52cb2769bb","Type":"ContainerStarted","Data":"7ae82469085929472726d10c20baf10affcd0edcef982610bf97634aab322c92"} Nov 02 17:53:22.929138 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.928989 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" event={"ID":"15a19313-0559-4a24-956b-2b52cb2769bb","Type":"ContainerStarted","Data":"85c0da5fe0e942f202f124d03854a1b9c6300647ab66b50570b2a89b1de21587"} Nov 02 17:53:22.960364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.960268 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" podStartSLOduration=1.960248851 podStartE2EDuration="1.960248851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:22.956795417 +0000 UTC m=+800.399878056" watchObservedRunningTime="2026-11-02 17:53:22.960248851 +0000 UTC m=+800.403331488" Nov 02 17:53:22.994821 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.994774 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8abf3330-c4x5p\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:22.995012 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.994849 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8abf3330-c4x5p\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:22.995499 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:22.995474 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxc8\" (UniqueName: \"kubernetes.io/projected/a919f38e-54f3-428a-8284-9aadff12c1c7-kube-api-access-ldxc8\") pod \"maas-api-revoke-sub-8abf3330-c4x5p\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:23.030805 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:23.030774 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557797b3_5dcf_4373_8a05_a8aa9fba451e.slice/crio-f8f9e0b7286b96202b2aa13bd59e9aee3931795928ff73ff85bdd38202aab822 WatchSource:0}: Error finding container f8f9e0b7286b96202b2aa13bd59e9aee3931795928ff73ff85bdd38202aab822: Status 404 returned error can't find the container with id f8f9e0b7286b96202b2aa13bd59e9aee3931795928ff73ff85bdd38202aab822 Nov 02 17:53:23.032659 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.032630 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk"] Nov 02 17:53:23.033754 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.033719 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:23.033849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.033781 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:23.033849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.033813 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:23.097502 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.097478 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8abf3330-c4x5p\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:23.097644 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.097537 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8abf3330-c4x5p\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:23.097753 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.097738 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxc8\" (UniqueName: \"kubernetes.io/projected/a919f38e-54f3-428a-8284-9aadff12c1c7-kube-api-access-ldxc8\") pod \"maas-api-revoke-sub-8abf3330-c4x5p\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:23.098240 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.098216 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8abf3330-c4x5p\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:23.100169 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.100146 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8abf3330-c4x5p\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:23.109530 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.109510 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxc8\" (UniqueName: \"kubernetes.io/projected/a919f38e-54f3-428a-8284-9aadff12c1c7-kube-api-access-ldxc8\") pod \"maas-api-revoke-sub-8abf3330-c4x5p\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:23.171894 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.170633 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:23.306253 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.305974 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:23.502004 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:23.501969 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda919f38e_54f3_428a_8284_9aadff12c1c7.slice/crio-bd7b35cc203fc11419095312b67312e490d87feda3863d9d1fbaad9c66763662 WatchSource:0}: Error finding container bd7b35cc203fc11419095312b67312e490d87feda3863d9d1fbaad9c66763662: Status 404 returned error can't find the container with id bd7b35cc203fc11419095312b67312e490d87feda3863d9d1fbaad9c66763662 Nov 02 17:53:23.504559 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.504530 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p"] Nov 02 17:53:23.940004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.939903 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" event={"ID":"557797b3-5dcf-4373-8a05-a8aa9fba451e","Type":"ContainerStarted","Data":"48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424"} Nov 02 17:53:23.940004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.939968 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" event={"ID":"557797b3-5dcf-4373-8a05-a8aa9fba451e","Type":"ContainerStarted","Data":"f8f9e0b7286b96202b2aa13bd59e9aee3931795928ff73ff85bdd38202aab822"} Nov 02 17:53:23.941766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.941728 4310 generic.go:356] "Generic (PLEG): container finished" podID="15a19313-0559-4a24-956b-2b52cb2769bb" containerID="7ae82469085929472726d10c20baf10affcd0edcef982610bf97634aab322c92" exitCode=0 Nov 02 17:53:23.942013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.941835 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" event={"ID":"15a19313-0559-4a24-956b-2b52cb2769bb","Type":"ContainerDied","Data":"7ae82469085929472726d10c20baf10affcd0edcef982610bf97634aab322c92"} Nov 02 17:53:23.967049 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.967002 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" event={"ID":"a919f38e-54f3-428a-8284-9aadff12c1c7","Type":"ContainerStarted","Data":"2208095a031039f93d088092e965bad278b28bf96423c1ccdf82102f92df6a29"} Nov 02 17:53:23.967376 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.967105 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" event={"ID":"a919f38e-54f3-428a-8284-9aadff12c1c7","Type":"ContainerStarted","Data":"bd7b35cc203fc11419095312b67312e490d87feda3863d9d1fbaad9c66763662"} Nov 02 17:53:23.996069 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:23.996016 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" podStartSLOduration=1.9960009570000001 podStartE2EDuration="1.996000957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:23.990071546 +0000 UTC m=+801.433154193" watchObservedRunningTime="2026-11-02 17:53:23.996000957 +0000 UTC m=+801.439083593" Nov 02 17:53:24.039005 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:24.038944 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" podStartSLOduration=2.038929105 podStartE2EDuration="2.038929105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:24.032282568 +0000 UTC m=+801.475365203" watchObservedRunningTime="2026-11-02 17:53:24.038929105 +0000 UTC m=+801.482011764" Nov 02 17:53:24.392102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:24.387978 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:24.728380 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:24.728350 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:24.729628 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:24.729606 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:53:24.729679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:24.729656 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:53:24.977595 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:24.977558 4310 generic.go:356] "Generic (PLEG): container finished" podID="a919f38e-54f3-428a-8284-9aadff12c1c7" containerID="2208095a031039f93d088092e965bad278b28bf96423c1ccdf82102f92df6a29" exitCode=0 Nov 02 17:53:24.977743 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:24.977616 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" event={"ID":"a919f38e-54f3-428a-8284-9aadff12c1c7","Type":"ContainerDied","Data":"2208095a031039f93d088092e965bad278b28bf96423c1ccdf82102f92df6a29"} Nov 02 17:53:25.183532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.183501 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl"] Nov 02 17:53:25.190911 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.190885 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl"] Nov 02 17:53:25.191066 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.191050 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:25.194131 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.193992 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:25.287596 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.287487 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6cgk\" (UniqueName: \"kubernetes.io/projected/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-kube-api-access-h6cgk\") pod \"maas-api-revoke-sub-8b316709-5qldl\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:25.287857 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.287817 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b316709-5qldl\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:25.288154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.288127 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b316709-5qldl\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:25.389098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.389056 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-openshift-ca\") pod \"15a19313-0559-4a24-956b-2b52cb2769bb\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " Nov 02 17:53:25.389281 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.389172 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a19313-0559-4a24-956b-2b52cb2769bb-kube-api-access-s2vcm\" (UniqueName: \"kubernetes.io/projected/15a19313-0559-4a24-956b-2b52cb2769bb-kube-api-access-s2vcm\") pod \"15a19313-0559-4a24-956b-2b52cb2769bb\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " Nov 02 17:53:25.389338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.389321 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-xks-ca\") pod \"15a19313-0559-4a24-956b-2b52cb2769bb\" (UID: \"15a19313-0559-4a24-956b-2b52cb2769bb\") " Nov 02 17:53:25.389479 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.389460 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-openshift-ca" pod "15a19313-0559-4a24-956b-2b52cb2769bb" (UID: "15a19313-0559-4a24-956b-2b52cb2769bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:25.389733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.389708 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b316709-5qldl\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:25.389969 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.389951 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b316709-5qldl\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:25.390118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.390012 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6cgk\" (UniqueName: \"kubernetes.io/projected/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-kube-api-access-h6cgk\") pod \"maas-api-revoke-sub-8b316709-5qldl\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:25.390339 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.390319 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.390696 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.390670 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b316709-5qldl\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:25.391395 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.391374 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-xks-ca" pod "15a19313-0559-4a24-956b-2b52cb2769bb" (UID: "15a19313-0559-4a24-956b-2b52cb2769bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:25.391723 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.391702 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a19313-0559-4a24-956b-2b52cb2769bb-kube-api-access-s2vcm" pod "15a19313-0559-4a24-956b-2b52cb2769bb" (UID: "15a19313-0559-4a24-956b-2b52cb2769bb"). InnerVolumeSpecName "kube-api-access-s2vcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:25.392370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.392349 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b316709-5qldl\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:25.401727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.401708 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6cgk\" (UniqueName: \"kubernetes.io/projected/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-kube-api-access-h6cgk\") pod \"maas-api-revoke-sub-8b316709-5qldl\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:25.433338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.433311 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:25.499163 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.492587 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a19313-0559-4a24-956b-2b52cb2769bb-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.499163 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.492631 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2vcm\" (UniqueName: \"kubernetes.io/projected/15a19313-0559-4a24-956b-2b52cb2769bb-kube-api-access-s2vcm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:25.510387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.510346 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:25.728702 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.728670 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:53:25.729265 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.729223 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:53:25.926671 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:25.926640 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf2345b_ecfe_4ac8_8d4a_e9d07c495e35.slice/crio-a9f3863d7f814969ff88f9d4cfdc39bba23ba759a6e3b07d967ae62dee16a48d WatchSource:0}: Error finding container a9f3863d7f814969ff88f9d4cfdc39bba23ba759a6e3b07d967ae62dee16a48d: Status 404 returned error can't find the container with id a9f3863d7f814969ff88f9d4cfdc39bba23ba759a6e3b07d967ae62dee16a48d Nov 02 17:53:25.928038 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.928013 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl"] Nov 02 17:53:25.992974 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.992943 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" Nov 02 17:53:25.993139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.993040 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv" event={"ID":"15a19313-0559-4a24-956b-2b52cb2769bb","Type":"ContainerDied","Data":"85c0da5fe0e942f202f124d03854a1b9c6300647ab66b50570b2a89b1de21587"} Nov 02 17:53:25.993191 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.993147 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85c0da5fe0e942f202f124d03854a1b9c6300647ab66b50570b2a89b1de21587" Nov 02 17:53:25.996254 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:25.996231 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" event={"ID":"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35","Type":"ContainerStarted","Data":"a9f3863d7f814969ff88f9d4cfdc39bba23ba759a6e3b07d967ae62dee16a48d"} Nov 02 17:53:26.218719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.218692 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:26.376108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.369411 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:26.417740 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.417695 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-xks-ca\") pod \"a919f38e-54f3-428a-8284-9aadff12c1c7\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " Nov 02 17:53:26.418010 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.417987 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a919f38e-54f3-428a-8284-9aadff12c1c7-kube-api-access-ldxc8\" (UniqueName: \"kubernetes.io/projected/a919f38e-54f3-428a-8284-9aadff12c1c7-kube-api-access-ldxc8\") pod \"a919f38e-54f3-428a-8284-9aadff12c1c7\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " Nov 02 17:53:26.418534 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.418507 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-openshift-ca\") pod \"a919f38e-54f3-428a-8284-9aadff12c1c7\" (UID: \"a919f38e-54f3-428a-8284-9aadff12c1c7\") " Nov 02 17:53:26.428325 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.428287 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-openshift-ca" pod "a919f38e-54f3-428a-8284-9aadff12c1c7" (UID: "a919f38e-54f3-428a-8284-9aadff12c1c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:26.430659 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.430632 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:26.454986 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.454942 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a919f38e-54f3-428a-8284-9aadff12c1c7-kube-api-access-ldxc8" pod "a919f38e-54f3-428a-8284-9aadff12c1c7" (UID: "a919f38e-54f3-428a-8284-9aadff12c1c7"). InnerVolumeSpecName "kube-api-access-ldxc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:26.465254 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.465196 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-xks-ca" pod "a919f38e-54f3-428a-8284-9aadff12c1c7" (UID: "a919f38e-54f3-428a-8284-9aadff12c1c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:26.534154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.533873 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a919f38e-54f3-428a-8284-9aadff12c1c7-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:26.534154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.533913 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldxc8\" (UniqueName: \"kubernetes.io/projected/a919f38e-54f3-428a-8284-9aadff12c1c7-kube-api-access-ldxc8\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:26.728432 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.728406 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:53:26.728578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:26.728456 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:53:27.006483 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:27.006397 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" event={"ID":"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35","Type":"ContainerStarted","Data":"eccab6c439029c57d1a61d1d86335b40c67a1f23795340aebf39eb0c5a929ff4"} Nov 02 17:53:27.007907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:27.007888 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" Nov 02 17:53:27.007907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:27.007896 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p" event={"ID":"a919f38e-54f3-428a-8284-9aadff12c1c7","Type":"ContainerDied","Data":"bd7b35cc203fc11419095312b67312e490d87feda3863d9d1fbaad9c66763662"} Nov 02 17:53:27.008106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:27.007934 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd7b35cc203fc11419095312b67312e490d87feda3863d9d1fbaad9c66763662" Nov 02 17:53:27.038910 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:27.038863 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" podStartSLOduration=2.038848858 podStartE2EDuration="2.038848858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:27.032768119 +0000 UTC m=+804.475850754" watchObservedRunningTime="2026-11-02 17:53:27.038848858 +0000 UTC m=+804.481931494" Nov 02 17:53:27.377379 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:27.377231 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:27.728062 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:27.728033 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:53:27.728230 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:27.728120 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:53:28.017762 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:28.017675 4310 generic.go:356] "Generic (PLEG): container finished" podID="faf2345b-ecfe-4ac8-8d4a-e9d07c495e35" containerID="eccab6c439029c57d1a61d1d86335b40c67a1f23795340aebf39eb0c5a929ff4" exitCode=0 Nov 02 17:53:28.018136 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:28.017764 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" event={"ID":"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35","Type":"ContainerDied","Data":"eccab6c439029c57d1a61d1d86335b40c67a1f23795340aebf39eb0c5a929ff4"} Nov 02 17:53:28.283053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:28.282947 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:28.728764 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:28.728729 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:53:28.728949 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:28.728785 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:53:29.265098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.254032 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:29.355866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.355792 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:29.441823 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.441769 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-xks-ca\") pod \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " Nov 02 17:53:29.442028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.441888 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-kube-api-access-h6cgk\" (UniqueName: \"kubernetes.io/projected/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-kube-api-access-h6cgk\") pod \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " Nov 02 17:53:29.442165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.442139 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-openshift-ca\") pod \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\" (UID: \"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35\") " Nov 02 17:53:29.442666 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.442626 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-openshift-ca" pod "faf2345b-ecfe-4ac8-8d4a-e9d07c495e35" (UID: "faf2345b-ecfe-4ac8-8d4a-e9d07c495e35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:29.442981 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.442948 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:29.444732 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.444703 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-xks-ca" pod "faf2345b-ecfe-4ac8-8d4a-e9d07c495e35" (UID: "faf2345b-ecfe-4ac8-8d4a-e9d07c495e35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:29.445306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.445278 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-kube-api-access-h6cgk" pod "faf2345b-ecfe-4ac8-8d4a-e9d07c495e35" (UID: "faf2345b-ecfe-4ac8-8d4a-e9d07c495e35"). InnerVolumeSpecName "kube-api-access-h6cgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:29.545206 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.545169 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:29.545206 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.545202 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6cgk\" (UniqueName: \"kubernetes.io/projected/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35-kube-api-access-h6cgk\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:29.728643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.728602 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:53:29.728821 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:29.728669 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 02 17:53:30.035480 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:30.035384 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" Nov 02 17:53:30.035659 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:30.035465 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl" event={"ID":"faf2345b-ecfe-4ac8-8d4a-e9d07c495e35","Type":"ContainerDied","Data":"a9f3863d7f814969ff88f9d4cfdc39bba23ba759a6e3b07d967ae62dee16a48d"} Nov 02 17:53:30.035659 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:30.035527 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9f3863d7f814969ff88f9d4cfdc39bba23ba759a6e3b07d967ae62dee16a48d" Nov 02 17:53:30.111143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:30.111110 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:30.763452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:30.763321 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:30.764514 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:30.764489 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:30.765456 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:30.765434 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:53:30.951264 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:30.951230 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:31.780472 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:31.780424 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:32.694581 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:32.694547 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:33.524210 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:53:33.524175 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:53:33.546600 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:33.546564 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:34.297515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:34.297471 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:35.272494 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:35.272460 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:36.064476 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.064438 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:36.211841 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.211795 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v"] Nov 02 17:53:36.213297 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.213236 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a919f38e-54f3-428a-8284-9aadff12c1c7" containerName="revoke-keys" Nov 02 17:53:36.213515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.213489 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a919f38e-54f3-428a-8284-9aadff12c1c7" containerName="revoke-keys" Nov 02 17:53:36.213711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.213698 4310 cpu_manager.go:414] "containerMap: removing container" podUID="15a19313-0559-4a24-956b-2b52cb2769bb" containerName="revoke-keys" Nov 02 17:53:36.213917 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.213903 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a19313-0559-4a24-956b-2b52cb2769bb" containerName="revoke-keys" Nov 02 17:53:36.214013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.214003 4310 cpu_manager.go:414] "containerMap: removing container" podUID="faf2345b-ecfe-4ac8-8d4a-e9d07c495e35" containerName="revoke-keys" Nov 02 17:53:36.214121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.214109 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf2345b-ecfe-4ac8-8d4a-e9d07c495e35" containerName="revoke-keys" Nov 02 17:53:36.214409 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.214384 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a19313-0559-4a24-956b-2b52cb2769bb" containerName="revoke-keys" Nov 02 17:53:36.214533 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.214522 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf2345b-ecfe-4ac8-8d4a-e9d07c495e35" containerName="revoke-keys" Nov 02 17:53:36.214630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.214618 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a919f38e-54f3-428a-8284-9aadff12c1c7" containerName="revoke-keys" Nov 02 17:53:36.220396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.220372 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:36.228001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.227973 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v"] Nov 02 17:53:36.240103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.237071 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:53:36.240103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.239755 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:53:36.344689 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.344590 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1217674-6sx2v\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:36.344689 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.344631 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1217674-6sx2v\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:36.345259 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.344723 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbjs\" (UniqueName: \"kubernetes.io/projected/4d71e3bb-0fb8-4a62-9e43-36bff691191f-kube-api-access-mnbjs\") pod \"maas-api-revoke-sub-e1217674-6sx2v\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:36.446371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.446339 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1217674-6sx2v\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:36.446577 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.446403 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1217674-6sx2v\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:36.446577 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.446462 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbjs\" (UniqueName: \"kubernetes.io/projected/4d71e3bb-0fb8-4a62-9e43-36bff691191f-kube-api-access-mnbjs\") pod \"maas-api-revoke-sub-e1217674-6sx2v\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:36.447116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.447065 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1217674-6sx2v\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:36.448841 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.448820 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1217674-6sx2v\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:36.463095 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.463062 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbjs\" (UniqueName: \"kubernetes.io/projected/4d71e3bb-0fb8-4a62-9e43-36bff691191f-kube-api-access-mnbjs\") pod \"maas-api-revoke-sub-e1217674-6sx2v\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:36.526568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.526530 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5"] Nov 02 17:53:36.531499 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.531478 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:36.553149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.548023 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c154cd4e-mj7j5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:36.553149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.548141 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c154cd4e-mj7j5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:36.553149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.548266 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhqr6\" (UniqueName: \"kubernetes.io/projected/eafe078b-f2a5-4333-aedc-10ba3c204fa5-kube-api-access-hhqr6\") pod \"maas-api-revoke-sub-c154cd4e-mj7j5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:36.553149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.550663 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5"] Nov 02 17:53:36.558784 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.558752 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:36.651300 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.651201 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqr6\" (UniqueName: \"kubernetes.io/projected/eafe078b-f2a5-4333-aedc-10ba3c204fa5-kube-api-access-hhqr6\") pod \"maas-api-revoke-sub-c154cd4e-mj7j5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:36.652741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.652709 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c154cd4e-mj7j5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:36.652870 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.652843 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c154cd4e-mj7j5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:36.655500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.653758 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c154cd4e-mj7j5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:36.656514 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.656455 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c154cd4e-mj7j5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:36.692954 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.691668 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqr6\" (UniqueName: \"kubernetes.io/projected/eafe078b-f2a5-4333-aedc-10ba3c204fa5-kube-api-access-hhqr6\") pod \"maas-api-revoke-sub-c154cd4e-mj7j5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:36.851870 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.851837 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:36.894401 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:36.894373 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d71e3bb_0fb8_4a62_9e43_36bff691191f.slice/crio-3a859b91f080c2121f5f2d84473e223382a7d8b0c4e1de2896e14372a1798b2e WatchSource:0}: Error finding container 3a859b91f080c2121f5f2d84473e223382a7d8b0c4e1de2896e14372a1798b2e: Status 404 returned error can't find the container with id 3a859b91f080c2121f5f2d84473e223382a7d8b0c4e1de2896e14372a1798b2e Nov 02 17:53:36.896924 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:36.896897 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v"] Nov 02 17:53:37.146559 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:37.146524 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafe078b_f2a5_4333_aedc_10ba3c204fa5.slice/crio-1ec597b26ffdfd31df6a53176f5e7253234d7244c5aced40774333c454a4fd93 WatchSource:0}: Error finding container 1ec597b26ffdfd31df6a53176f5e7253234d7244c5aced40774333c454a4fd93: Status 404 returned error can't find the container with id 1ec597b26ffdfd31df6a53176f5e7253234d7244c5aced40774333c454a4fd93 Nov 02 17:53:37.149525 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:37.148850 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5"] Nov 02 17:53:37.162715 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:37.162680 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" event={"ID":"eafe078b-f2a5-4333-aedc-10ba3c204fa5","Type":"ContainerStarted","Data":"1ec597b26ffdfd31df6a53176f5e7253234d7244c5aced40774333c454a4fd93"} Nov 02 17:53:37.164490 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:37.164462 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" event={"ID":"4d71e3bb-0fb8-4a62-9e43-36bff691191f","Type":"ContainerStarted","Data":"3a859b91f080c2121f5f2d84473e223382a7d8b0c4e1de2896e14372a1798b2e"} Nov 02 17:53:37.183690 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:37.183659 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:38.042474 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.037149 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj"] Nov 02 17:53:38.098172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.083669 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:38.107578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.089611 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33d927b-jswxj\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:38.107578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.099190 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmjwg\" (UniqueName: \"kubernetes.io/projected/25f67fab-d093-4478-a09b-8d6e2921d0c0-kube-api-access-gmjwg\") pod \"maas-api-revoke-sub-b33d927b-jswxj\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:38.107578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.099432 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33d927b-jswxj\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:38.240103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.201412 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33d927b-jswxj\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:38.348109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.346106 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmjwg\" (UniqueName: \"kubernetes.io/projected/25f67fab-d093-4478-a09b-8d6e2921d0c0-kube-api-access-gmjwg\") pod \"maas-api-revoke-sub-b33d927b-jswxj\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:38.348109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.346839 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33d927b-jswxj\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:38.348346 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.348207 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33d927b-jswxj\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:38.570156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.558537 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj"] Nov 02 17:53:38.711103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.710472 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:38.726108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.725180 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:38.775957 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.775624 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:38.906275 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.889220 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:53:38.955100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.922908 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33d927b-jswxj\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:38.979109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.964400 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:53:39.128839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:38.969517 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:53:39.137187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.063581 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmjwg\" (UniqueName: \"kubernetes.io/projected/25f67fab-d093-4478-a09b-8d6e2921d0c0-kube-api-access-gmjwg\") pod \"maas-api-revoke-sub-b33d927b-jswxj\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:39.329098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.328171 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:39.329098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.328314 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:39.329438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.329411 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:39.389376 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.389328 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" event={"ID":"4d71e3bb-0fb8-4a62-9e43-36bff691191f","Type":"ContainerStarted","Data":"1d1100c9358a7e40b36c15c93483bae9ed77ddf3b72a8b6dfd2231ad16d0971e"} Nov 02 17:53:39.389829 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.389645 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" event={"ID":"eafe078b-f2a5-4333-aedc-10ba3c204fa5","Type":"ContainerStarted","Data":"b513e8dafdff4b6efe2ef9a9d4a9256992110379ee4dde01df9cc5ad72f25020"} Nov 02 17:53:39.413302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.412826 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:39.519131 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.515191 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" podStartSLOduration=3.515166935 podStartE2EDuration="3.515166935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:39.07841451 +0000 UTC m=+816.521497170" watchObservedRunningTime="2026-11-02 17:53:39.515166935 +0000 UTC m=+816.958249571" Nov 02 17:53:39.570573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.570497 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" podStartSLOduration=3.570464948 podStartE2EDuration="3.570464948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:39.453796811 +0000 UTC m=+816.896879450" watchObservedRunningTime="2026-11-02 17:53:39.570464948 +0000 UTC m=+817.013547584" Nov 02 17:53:39.723604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.712639 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:53:39.723604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.713257 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:53:39.723604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.714913 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:53:39.773720 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.771724 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:39.773720 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.772823 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:53:39.794173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.779200 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:39.794173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.779618 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:53:39.865881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.865836 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:39.902173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.893470 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:53:39.902173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:39.894261 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:53:40.136178 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.113333 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded" Nov 02 17:53:40.317184 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.312575 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:40.347259 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.345433 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:53:40.392968 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.348185 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:53:40.392968 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.348494 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:53:40.421784 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.418397 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:40.421784 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.420261 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:53:40.453310 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.453274 4310 generic.go:356] "Generic (PLEG): container finished" podID="4d71e3bb-0fb8-4a62-9e43-36bff691191f" containerID="1d1100c9358a7e40b36c15c93483bae9ed77ddf3b72a8b6dfd2231ad16d0971e" exitCode=0 Nov 02 17:53:40.453925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.453897 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" event={"ID":"4d71e3bb-0fb8-4a62-9e43-36bff691191f","Type":"ContainerDied","Data":"1d1100c9358a7e40b36c15c93483bae9ed77ddf3b72a8b6dfd2231ad16d0971e"} Nov 02 17:53:40.489565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.489527 4310 generic.go:356] "Generic (PLEG): container finished" podID="eafe078b-f2a5-4333-aedc-10ba3c204fa5" containerID="b513e8dafdff4b6efe2ef9a9d4a9256992110379ee4dde01df9cc5ad72f25020" exitCode=0 Nov 02 17:53:40.515652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.513400 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" event={"ID":"eafe078b-f2a5-4333-aedc-10ba3c204fa5","Type":"ContainerDied","Data":"b513e8dafdff4b6efe2ef9a9d4a9256992110379ee4dde01df9cc5ad72f25020"} Nov 02 17:53:40.784105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.766165 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:40.784105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.766453 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:53:40.825798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.800307 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:40.825798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.802139 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:40.889821 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.872654 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.019104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:40.982925 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.341290 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:41.316366 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.341290 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:41.323409 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:41.341290 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:41.323478 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:53:41.370991 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:41.370935 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.455437 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:41.455373 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.782441 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:41.782402 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:53:41.789260 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:41.789217 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:53:41.806268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:41.806211 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:41.807876 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:41.807828 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:53:41.880110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:41.877310 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.204153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:42.179346 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.204513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:42.204477 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:53:42.340977 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:42.340929 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.432386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:42.429138 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.532441 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:42.513260 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.882835 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:42.881196 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:42.911141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:42.909267 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:53:43.069392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.069344 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:53:43.078105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.070366 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.250166 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.206384 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:53:43.482103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.478630 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.738167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.489597 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-8twlt" podUID="1705d3f8-7617-44ed-b8c5-6adb2e14cb55" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.32:8443/ready\": context deadline exceeded" Nov 02 17:53:43.738167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.507343 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:53:43.738167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.583199 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.738167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.606979 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded" Nov 02 17:53:43.738167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.731623 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:53:43.738167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.736591 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" podUID="d4955081-1a71-4933-8332-dd8bc035ae18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.914106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.893095 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:43.985374 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:43.926302 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:53:44.092101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:44.092027 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" podUID="2144453b-de91-4404-8b54-5960b5a697ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:44.180326 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:44.092342 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:44.320164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:44.252392 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:53:44.769105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:44.092379 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:53:44.791110 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:53:44.785616 4310 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.903s" Nov 02 17:53:45.187163 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:45.186698 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" event={"ID":"25f67fab-d093-4478-a09b-8d6e2921d0c0","Type":"ContainerStarted","Data":"c319097831fcd5d96ca15d00e019bc186fa1e03a1bc4ca954fffc28de2b2a71e"} Nov 02 17:53:45.554489 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:45.534744 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Nov 02 17:53:45.580172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:45.555068 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.581583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:45.581532 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:53:45.603337 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:45.582188 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:53:45.603337 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:45.582477 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:45.603337 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:45.583070 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:53:45.603337 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:45.583345 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:53:45.781048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:45.619716 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.145165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.045796 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.145165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.045862 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" podUID="aed79116-9b99-4427-814f-e8af212f5126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.145165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.126489 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.184194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.148992 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.184194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.172347 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.184194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.172402 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.224105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.192845 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.224105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.192907 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.321552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.321516 4310 generic.go:356] "Generic (PLEG): container finished" podID="25f67fab-d093-4478-a09b-8d6e2921d0c0" containerID="a9ce481d303ddf83b00824f47e556499dc4fa7d4139f11f52037275d7ed39852" exitCode=0 Nov 02 17:53:46.321957 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.321927 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" event={"ID":"25f67fab-d093-4478-a09b-8d6e2921d0c0","Type":"ContainerDied","Data":"a9ce481d303ddf83b00824f47e556499dc4fa7d4139f11f52037275d7ed39852"} Nov 02 17:53:46.372644 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.372596 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" event={"ID":"4d71e3bb-0fb8-4a62-9e43-36bff691191f","Type":"ContainerDied","Data":"3a859b91f080c2121f5f2d84473e223382a7d8b0c4e1de2896e14372a1798b2e"} Nov 02 17:53:46.412617 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.406006 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a859b91f080c2121f5f2d84473e223382a7d8b0c4e1de2896e14372a1798b2e" Nov 02 17:53:46.412617 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.389406 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.412617 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.408206 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Nov 02 17:53:46.412617 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.410131 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.413844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.413054 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.413844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.413123 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.432331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.432299 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:46.451700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.451661 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.451976 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.451951 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.543281 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.543246 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:46.543594 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.543548 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.553107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.552524 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d71e3bb-0fb8-4a62-9e43-36bff691191f-kube-api-access-mnbjs\" (UniqueName: \"kubernetes.io/projected/4d71e3bb-0fb8-4a62-9e43-36bff691191f-kube-api-access-mnbjs\") pod \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " Nov 02 17:53:46.603007 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.597365 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:53:46.623103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.611384 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:53:46.645249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.645199 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.670271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.646558 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:46.716107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.661581 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:53:46.743455 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.661865 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:53:46.744142 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.665400 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:53:46.855203 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.855149 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:47.001647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.001605 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:47.001950 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.958452 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:53:47.003553 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.003529 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:47.053126 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.050620 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d71e3bb-0fb8-4a62-9e43-36bff691191f-kube-api-access-mnbjs" pod "4d71e3bb-0fb8-4a62-9e43-36bff691191f" (UID: "4d71e3bb-0fb8-4a62-9e43-36bff691191f"). InnerVolumeSpecName "kube-api-access-mnbjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:47.081440 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.081376 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:47.104327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.104274 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:47.226337 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.094435 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:47.290754 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.168741 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:47.291038 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.290996 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:47.322486 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.322431 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:47.413154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.413104 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-xks-ca" pod "4d71e3bb-0fb8-4a62-9e43-36bff691191f" (UID: "4d71e3bb-0fb8-4a62-9e43-36bff691191f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:47.413523 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.413478 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:53:47.414184 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.414141 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:47.452232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.425743 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:47.598785 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.598684 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:47.602191 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:46.560433 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-xks-ca\") pod \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " Nov 02 17:53:47.606100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.602443 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-openshift-ca\") pod \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " Nov 02 17:53:47.632106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.606660 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafe078b-f2a5-4333-aedc-10ba3c204fa5-kube-api-access-hhqr6\" (UniqueName: \"kubernetes.io/projected/eafe078b-f2a5-4333-aedc-10ba3c204fa5-kube-api-access-hhqr6\") pod \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " Nov 02 17:53:47.644028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.484175 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:47.711574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.711523 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:47.773246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.655359 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-openshift-ca\") pod \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\" (UID: \"4d71e3bb-0fb8-4a62-9e43-36bff691191f\") " Nov 02 17:53:47.840306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.627721 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-openshift-ca" pod "eafe078b-f2a5-4333-aedc-10ba3c204fa5" (UID: "eafe078b-f2a5-4333-aedc-10ba3c204fa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:47.891213 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.711421 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-openshift-ca" pod "4d71e3bb-0fb8-4a62-9e43-36bff691191f" (UID: "4d71e3bb-0fb8-4a62-9e43-36bff691191f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:47.891213 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.888737 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:47.891213 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.888769 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:47.891213 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.888793 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d71e3bb-0fb8-4a62-9e43-36bff691191f-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:47.891213 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.888809 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnbjs\" (UniqueName: \"kubernetes.io/projected/4d71e3bb-0fb8-4a62-9e43-36bff691191f-kube-api-access-mnbjs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:48.006495 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:47.969835 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafe078b-f2a5-4333-aedc-10ba3c204fa5-kube-api-access-hhqr6" pod "eafe078b-f2a5-4333-aedc-10ba3c204fa5" (UID: "eafe078b-f2a5-4333-aedc-10ba3c204fa5"). InnerVolumeSpecName "kube-api-access-hhqr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:48.043183 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.018479 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:48.043530 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.043479 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:48.047682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.047648 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-xks-ca\") pod \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\" (UID: \"eafe078b-f2a5-4333-aedc-10ba3c204fa5\") " Nov 02 17:53:48.064218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.064180 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhqr6\" (UniqueName: \"kubernetes.io/projected/eafe078b-f2a5-4333-aedc-10ba3c204fa5-kube-api-access-hhqr6\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:48.134103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.122885 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:53:48.143606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.143509 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:53:48.181106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.163769 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:48.210680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.182309 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:53:48.210937 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.182808 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:53:48.257137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.182855 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:53:48.257137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.184136 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:53:48.257137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.210296 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-xks-ca" pod "eafe078b-f2a5-4333-aedc-10ba3c204fa5" (UID: "eafe078b-f2a5-4333-aedc-10ba3c204fa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:48.257137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.253007 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:48.257137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.254458 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:48.305241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.305174 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:48.328605 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.309849 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:48.328605 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.312067 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:48.331412 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.331373 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafe078b-f2a5-4333-aedc-10ba3c204fa5-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:48.360705 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.360664 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:48.361004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.360973 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:48.401153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.397089 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v" Nov 02 17:53:48.401543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.401332 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" Nov 02 17:53:48.423414 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:53:48.423361 4310 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.508s" Nov 02 17:53:48.452106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.427451 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5" event={"ID":"eafe078b-f2a5-4333-aedc-10ba3c204fa5","Type":"ContainerDied","Data":"1ec597b26ffdfd31df6a53176f5e7253234d7244c5aced40774333c454a4fd93"} Nov 02 17:53:48.452399 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.452373 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ec597b26ffdfd31df6a53176f5e7253234d7244c5aced40774333c454a4fd93" Nov 02 17:53:48.456029 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.452515 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj"] Nov 02 17:53:48.488482 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.473233 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:53:48.488697 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.473413 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:48.550103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.522235 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:48.550103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.522325 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:48.550103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.522467 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:48.550103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:48.522492 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:49.125448 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.125318 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:49.171437 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.163832 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:49.171437 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.163938 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:49.240205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.229980 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:49.240205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.238157 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:49.240205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.238565 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:49.240205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.238854 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:53:49.254307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.254271 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:49.254957 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.254920 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:53:49.433165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.408361 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6"] Nov 02 17:53:49.548767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.548733 4310 cpu_manager.go:414] "containerMap: removing container" podUID="4d71e3bb-0fb8-4a62-9e43-36bff691191f" containerName="revoke-keys" Nov 02 17:53:49.555147 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.548956 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d71e3bb-0fb8-4a62-9e43-36bff691191f" containerName="revoke-keys" Nov 02 17:53:49.559168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.559138 4310 cpu_manager.go:414] "containerMap: removing container" podUID="eafe078b-f2a5-4333-aedc-10ba3c204fa5" containerName="revoke-keys" Nov 02 17:53:49.584161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.584119 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafe078b-f2a5-4333-aedc-10ba3c204fa5" containerName="revoke-keys" Nov 02 17:53:49.584681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.584660 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafe078b-f2a5-4333-aedc-10ba3c204fa5" containerName="revoke-keys" Nov 02 17:53:49.584844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.584832 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d71e3bb-0fb8-4a62-9e43-36bff691191f" containerName="revoke-keys" Nov 02 17:53:49.622425 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.622385 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:53:49.654481 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.654422 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:49.704987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.689290 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6"] Nov 02 17:53:49.704987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:49.693139 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:50.029105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.001055 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6lfc\" (UniqueName: \"kubernetes.io/projected/6e642a3b-628a-4120-8db7-77005882e0af-kube-api-access-t6lfc\") pod \"maas-api-revoke-sub-9942a114-gfsk6\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:50.045104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.037395 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9942a114-gfsk6\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:50.085110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.072903 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9942a114-gfsk6\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:50.188442 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.188395 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:50.294259 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.291757 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:53:50.308717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.305203 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:50.311260 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.311222 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded" Nov 02 17:53:50.368052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.353673 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:50.519386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.504706 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:53:50.519386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.505480 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:53:50.519386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.506257 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:53:50.568161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.560766 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9942a114-gfsk6\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:50.568161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.561061 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6lfc\" (UniqueName: \"kubernetes.io/projected/6e642a3b-628a-4120-8db7-77005882e0af-kube-api-access-t6lfc\") pod \"maas-api-revoke-sub-9942a114-gfsk6\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:50.568161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.561376 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9942a114-gfsk6\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:50.568161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.562510 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9942a114-gfsk6\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:50.668104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:50.657505 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:51.119945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.119888 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:51.179398 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:53:51.175558 4310 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafe078b_f2a5_4333_aedc_10ba3c204fa5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d71e3bb_0fb8_4a62_9e43_36bff691191f.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:53:51.202235 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.202185 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:51.221680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.221638 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9942a114-gfsk6\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:51.232346 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.232315 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52"] Nov 02 17:53:51.268351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.268325 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:53:51.268812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.268794 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:51.270900 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.270882 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:51.273568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.268616 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52"] Nov 02 17:53:51.273683 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.273655 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:53:51.273744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.273716 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:53:51.273795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.273765 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:53:51.285954 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.285922 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" event={"ID":"25f67fab-d093-4478-a09b-8d6e2921d0c0","Type":"ContainerDied","Data":"c319097831fcd5d96ca15d00e019bc186fa1e03a1bc4ca954fffc28de2b2a71e"} Nov 02 17:53:51.286171 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.286148 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c319097831fcd5d96ca15d00e019bc186fa1e03a1bc4ca954fffc28de2b2a71e" Nov 02 17:53:51.294746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.294069 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:51.294746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.294552 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6lfc\" (UniqueName: \"kubernetes.io/projected/6e642a3b-628a-4120-8db7-77005882e0af-kube-api-access-t6lfc\") pod \"maas-api-revoke-sub-9942a114-gfsk6\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:51.296165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.296141 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:51.297112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.297059 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f67fab-d093-4478-a09b-8d6e2921d0c0-kube-api-access-gmjwg\" (UniqueName: \"kubernetes.io/projected/25f67fab-d093-4478-a09b-8d6e2921d0c0-kube-api-access-gmjwg\") pod \"25f67fab-d093-4478-a09b-8d6e2921d0c0\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " Nov 02 17:53:51.299011 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.298976 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-openshift-ca\") pod \"25f67fab-d093-4478-a09b-8d6e2921d0c0\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " Nov 02 17:53:51.306443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.306413 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-xks-ca\") pod \"25f67fab-d093-4478-a09b-8d6e2921d0c0\" (UID: \"25f67fab-d093-4478-a09b-8d6e2921d0c0\") " Nov 02 17:53:51.307435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.307401 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16e4609-wzp52\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:51.307694 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.307674 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16e4609-wzp52\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:51.307939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.307917 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn4n7\" (UniqueName: \"kubernetes.io/projected/8ba6d31f-770c-4907-84e9-ce759ff12719-kube-api-access-xn4n7\") pod \"maas-api-revoke-sub-e16e4609-wzp52\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:51.312361 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.306184 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-openshift-ca" pod "25f67fab-d093-4478-a09b-8d6e2921d0c0" (UID: "25f67fab-d093-4478-a09b-8d6e2921d0c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:51.322760 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.321772 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f67fab-d093-4478-a09b-8d6e2921d0c0-kube-api-access-gmjwg" pod "25f67fab-d093-4478-a09b-8d6e2921d0c0" (UID: "25f67fab-d093-4478-a09b-8d6e2921d0c0"). InnerVolumeSpecName "kube-api-access-gmjwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:51.323984 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.323382 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-xks-ca" pod "25f67fab-d093-4478-a09b-8d6e2921d0c0" (UID: "25f67fab-d093-4478-a09b-8d6e2921d0c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:51.337153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.336548 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:51.337153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.336580 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmjwg\" (UniqueName: \"kubernetes.io/projected/25f67fab-d093-4478-a09b-8d6e2921d0c0-kube-api-access-gmjwg\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:51.337153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.336656 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f67fab-d093-4478-a09b-8d6e2921d0c0-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:51.354953 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.346928 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:53:51.354953 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.351884 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:53:51.365314 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.360627 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:53:51.366478 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.366361 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:53:51.368563 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.368538 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:53:51.439368 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.439285 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16e4609-wzp52\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:51.439501 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.439404 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16e4609-wzp52\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:51.439569 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.439553 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn4n7\" (UniqueName: \"kubernetes.io/projected/8ba6d31f-770c-4907-84e9-ce759ff12719-kube-api-access-xn4n7\") pod \"maas-api-revoke-sub-e16e4609-wzp52\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:51.441442 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.441409 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16e4609-wzp52\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:51.443481 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.443441 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16e4609-wzp52\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:51.451636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.451610 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn4n7\" (UniqueName: \"kubernetes.io/projected/8ba6d31f-770c-4907-84e9-ce759ff12719-kube-api-access-xn4n7\") pod \"maas-api-revoke-sub-e16e4609-wzp52\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:51.621146 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.620961 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6"] Nov 02 17:53:51.622269 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:51.622131 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e642a3b_628a_4120_8db7_77005882e0af.slice/crio-f80fe7eb20f9bdda9f810eeb2987b288d58156db23facef452e357f63078eb6e WatchSource:0}: Error finding container f80fe7eb20f9bdda9f810eeb2987b288d58156db23facef452e357f63078eb6e: Status 404 returned error can't find the container with id f80fe7eb20f9bdda9f810eeb2987b288d58156db23facef452e357f63078eb6e Nov 02 17:53:51.626323 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.626296 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:51.793651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.793618 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:51.993804 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:51.987204 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52"] Nov 02 17:53:51.999958 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:51.999924 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba6d31f_770c_4907_84e9_ce759ff12719.slice/crio-a3d94915ac3cebdee712f93ed4f081f4c1499aafe92fe6f67892a455a1d69c22 WatchSource:0}: Error finding container a3d94915ac3cebdee712f93ed4f081f4c1499aafe92fe6f67892a455a1d69c22: Status 404 returned error can't find the container with id a3d94915ac3cebdee712f93ed4f081f4c1499aafe92fe6f67892a455a1d69c22 Nov 02 17:53:52.295109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.295039 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" event={"ID":"8ba6d31f-770c-4907-84e9-ce759ff12719","Type":"ContainerStarted","Data":"a3d94915ac3cebdee712f93ed4f081f4c1499aafe92fe6f67892a455a1d69c22"} Nov 02 17:53:52.296940 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.296911 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" event={"ID":"6e642a3b-628a-4120-8db7-77005882e0af","Type":"ContainerStarted","Data":"4fbf1f0d8b35beff7efb98441e0338b33b239d3e88c3d25da69c750ab23a1a8f"} Nov 02 17:53:52.297068 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.296963 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" event={"ID":"6e642a3b-628a-4120-8db7-77005882e0af","Type":"ContainerStarted","Data":"f80fe7eb20f9bdda9f810eeb2987b288d58156db23facef452e357f63078eb6e"} Nov 02 17:53:52.297068 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.296929 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj" Nov 02 17:53:52.325611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.325566 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" podStartSLOduration=6.325551755 podStartE2EDuration="6.325551755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:52.319731694 +0000 UTC m=+829.762814332" watchObservedRunningTime="2026-11-02 17:53:52.325551755 +0000 UTC m=+829.768634391" Nov 02 17:53:52.413206 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.413170 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg"] Nov 02 17:53:52.414253 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.414226 4310 cpu_manager.go:414] "containerMap: removing container" podUID="25f67fab-d093-4478-a09b-8d6e2921d0c0" containerName="revoke-keys" Nov 02 17:53:52.414430 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.414411 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f67fab-d093-4478-a09b-8d6e2921d0c0" containerName="revoke-keys" Nov 02 17:53:52.414773 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.414751 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f67fab-d093-4478-a09b-8d6e2921d0c0" containerName="revoke-keys" Nov 02 17:53:52.418834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.418811 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.439122 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.434300 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg"] Nov 02 17:53:52.577050 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.576970 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.577050 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.577019 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr44h\" (UniqueName: \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-kube-api-access-nr44h\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.577269 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.577237 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.577400 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.577381 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.577448 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.577417 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7f8807-ebff-4a43-bb70-a161a1df954b-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.577516 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.577496 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.577590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.577567 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.577643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.577591 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.577643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.577610 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.628776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.628740 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:52.693108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.688994 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.693108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.689067 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr44h\" (UniqueName: \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-kube-api-access-nr44h\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.693108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.689188 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.693108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.689334 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.693108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.689371 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7f8807-ebff-4a43-bb70-a161a1df954b-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.693108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.689428 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.693108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.689458 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.693108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.689482 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.693108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.689511 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.693108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.690019 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-credential-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.699862 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.699817 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-certs\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.726157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.712068 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7f8807-ebff-4a43-bb70-a161a1df954b-istiod-ca-cert\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.726157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.712373 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-socket\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.726157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.712591 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-data\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.726157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.721314 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-envoy\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.771920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.771620 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-token\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.863038 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.862949 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-podinfo\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.903152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.890858 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr44h\" (UniqueName: \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-kube-api-access-nr44h\") pod \"e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:52.906587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.906554 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng"] Nov 02 17:53:52.917828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.917796 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng"] Nov 02 17:53:52.917971 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.917899 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:52.961052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.961014 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:52.961282 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.961125 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:52.961282 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.961248 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhp4g\" (UniqueName: \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-kube-api-access-fhp4g\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:52.961385 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.961300 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:52.961444 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.961426 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e331e1-a05d-491b-a48c-913b3bb8f45e-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:52.961479 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.961460 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:52.961531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.961483 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:52.961602 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.961588 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:52.961646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:52.961634 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.042037 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.041998 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:53.063160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.063134 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e331e1-a05d-491b-a48c-913b3bb8f45e-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.063250 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.063185 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.063250 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.063225 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.063330 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.063300 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.063389 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.063371 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.063483 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.063469 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.063574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.063557 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.063660 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.063639 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhp4g\" (UniqueName: \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-kube-api-access-fhp4g\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.064450 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.064430 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.065118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.063850 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-credential-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.065118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.064015 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e331e1-a05d-491b-a48c-913b3bb8f45e-istiod-ca-cert\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.065295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.064057 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-socket\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.065295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.063752 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-data\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.065295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.064757 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-certs\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.066521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.066499 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-envoy\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.066635 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.066532 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-podinfo\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.072567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.072545 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhp4g\" (UniqueName: \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-kube-api-access-fhp4g\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.073294 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.073270 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-token\") pod \"e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.234851 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.234802 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:53.312524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.311141 4310 generic.go:356] "Generic (PLEG): container finished" podID="8ba6d31f-770c-4907-84e9-ce759ff12719" containerID="148c3b34ea16b536aa7bf837cb880cff23594109a1ec226cddb54a8d85174aff" exitCode=0 Nov 02 17:53:53.312524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.311406 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" event={"ID":"8ba6d31f-770c-4907-84e9-ce759ff12719","Type":"ContainerDied","Data":"148c3b34ea16b536aa7bf837cb880cff23594109a1ec226cddb54a8d85174aff"} Nov 02 17:53:53.314515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.314488 4310 generic.go:356] "Generic (PLEG): container finished" podID="6e642a3b-628a-4120-8db7-77005882e0af" containerID="4fbf1f0d8b35beff7efb98441e0338b33b239d3e88c3d25da69c750ab23a1a8f" exitCode=0 Nov 02 17:53:53.314661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.314591 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" event={"ID":"6e642a3b-628a-4120-8db7-77005882e0af","Type":"ContainerDied","Data":"4fbf1f0d8b35beff7efb98441e0338b33b239d3e88c3d25da69c750ab23a1a8f"} Nov 02 17:53:53.354939 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:53.354905 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7f8807_ebff_4a43_bb70_a161a1df954b.slice/crio-293400f2c8809c51f927cb83c20407cfa123735b92590140b0c7f3468a656c17 WatchSource:0}: Error finding container 293400f2c8809c51f927cb83c20407cfa123735b92590140b0c7f3468a656c17: Status 404 returned error can't find the container with id 293400f2c8809c51f927cb83c20407cfa123735b92590140b0c7f3468a656c17 Nov 02 17:53:53.358633 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.358602 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg"] Nov 02 17:53:53.359967 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.359933 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:53.360070 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.359998 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:53.360070 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.360032 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:53.430338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.430308 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:53.524056 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:53:53.524022 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e331e1_a05d_491b_a48c_913b3bb8f45e.slice/crio-10ee7b365f7722ad4238674de874adf741e137d97f70059c80601f992ac4d4c4 WatchSource:0}: Error finding container 10ee7b365f7722ad4238674de874adf741e137d97f70059c80601f992ac4d4c4: Status 404 returned error can't find the container with id 10ee7b365f7722ad4238674de874adf741e137d97f70059c80601f992ac4d4c4 Nov 02 17:53:53.525413 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.525389 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng"] Nov 02 17:53:53.527162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.527121 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:53.527247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.527193 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:53.527247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:53.527226 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:53:54.324618 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:54.324574 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" event={"ID":"58e331e1-a05d-491b-a48c-913b3bb8f45e","Type":"ContainerStarted","Data":"14715dacc085e0427bb077b4ae06842eaefb155c908617b2ad4650f36506c2df"} Nov 02 17:53:54.325019 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:54.324647 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" event={"ID":"58e331e1-a05d-491b-a48c-913b3bb8f45e","Type":"ContainerStarted","Data":"10ee7b365f7722ad4238674de874adf741e137d97f70059c80601f992ac4d4c4"} Nov 02 17:53:54.329125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:54.329067 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" event={"ID":"be7f8807-ebff-4a43-bb70-a161a1df954b","Type":"ContainerStarted","Data":"a63f05f51af7127c1bf44efb3a1ba75b49f13dff92b3e8d56e858af64a221702"} Nov 02 17:53:54.329285 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:54.329238 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" event={"ID":"be7f8807-ebff-4a43-bb70-a161a1df954b","Type":"ContainerStarted","Data":"293400f2c8809c51f927cb83c20407cfa123735b92590140b0c7f3468a656c17"} Nov 02 17:53:54.383100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:54.378858 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" podStartSLOduration=2.378836431 podStartE2EDuration="2.378836431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:54.359574549 +0000 UTC m=+831.802657212" watchObservedRunningTime="2026-11-02 17:53:54.378836431 +0000 UTC m=+831.821919067" Nov 02 17:53:54.413879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:54.413817 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" podStartSLOduration=2.4137929590000002 podStartE2EDuration="2.413792959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:54.396798727 +0000 UTC m=+831.839881362" watchObservedRunningTime="2026-11-02 17:53:54.413792959 +0000 UTC m=+831.856875596" Nov 02 17:53:54.461802 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:54.461696 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:55.012750 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.012719 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:55.036390 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.036357 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-xks-ca\") pod \"8ba6d31f-770c-4907-84e9-ce759ff12719\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " Nov 02 17:53:55.036587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.036530 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba6d31f-770c-4907-84e9-ce759ff12719-kube-api-access-xn4n7\" (UniqueName: \"kubernetes.io/projected/8ba6d31f-770c-4907-84e9-ce759ff12719-kube-api-access-xn4n7\") pod \"8ba6d31f-770c-4907-84e9-ce759ff12719\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " Nov 02 17:53:55.036662 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.036597 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-openshift-ca\") pod \"8ba6d31f-770c-4907-84e9-ce759ff12719\" (UID: \"8ba6d31f-770c-4907-84e9-ce759ff12719\") " Nov 02 17:53:55.037412 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.037388 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-openshift-ca" pod "8ba6d31f-770c-4907-84e9-ce759ff12719" (UID: "8ba6d31f-770c-4907-84e9-ce759ff12719"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:55.041152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.041072 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-xks-ca" pod "8ba6d31f-770c-4907-84e9-ce759ff12719" (UID: "8ba6d31f-770c-4907-84e9-ce759ff12719"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:55.054172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.054133 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:53:55.118389 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.118344 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:53:55.118582 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.118424 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba6d31f-770c-4907-84e9-ce759ff12719-kube-api-access-xn4n7" pod "8ba6d31f-770c-4907-84e9-ce759ff12719" (UID: "8ba6d31f-770c-4907-84e9-ce759ff12719"). InnerVolumeSpecName "kube-api-access-xn4n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:55.118582 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.118429 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 17:53:55.133522 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.133496 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:55.138766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.138743 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:55.138913 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.138902 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn4n7\" (UniqueName: \"kubernetes.io/projected/8ba6d31f-770c-4907-84e9-ce759ff12719-kube-api-access-xn4n7\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:55.138999 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.138990 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba6d31f-770c-4907-84e9-ce759ff12719-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:55.236086 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.236042 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:53:55.236937 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.236914 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:53:55.237129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.237108 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 17:53:55.240447 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.240425 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-openshift-ca\") pod \"6e642a3b-628a-4120-8db7-77005882e0af\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " Nov 02 17:53:55.240561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.240461 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e642a3b-628a-4120-8db7-77005882e0af-kube-api-access-t6lfc\" (UniqueName: \"kubernetes.io/projected/6e642a3b-628a-4120-8db7-77005882e0af-kube-api-access-t6lfc\") pod \"6e642a3b-628a-4120-8db7-77005882e0af\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " Nov 02 17:53:55.240727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.240710 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-xks-ca\") pod \"6e642a3b-628a-4120-8db7-77005882e0af\" (UID: \"6e642a3b-628a-4120-8db7-77005882e0af\") " Nov 02 17:53:55.240853 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.240826 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-openshift-ca" pod "6e642a3b-628a-4120-8db7-77005882e0af" (UID: "6e642a3b-628a-4120-8db7-77005882e0af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:55.241263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.241243 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:55.287273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.287233 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-xks-ca" pod "6e642a3b-628a-4120-8db7-77005882e0af" (UID: "6e642a3b-628a-4120-8db7-77005882e0af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:55.297725 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.297680 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e642a3b-628a-4120-8db7-77005882e0af-kube-api-access-t6lfc" pod "6e642a3b-628a-4120-8db7-77005882e0af" (UID: "6e642a3b-628a-4120-8db7-77005882e0af"). InnerVolumeSpecName "kube-api-access-t6lfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:55.338149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.338110 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" Nov 02 17:53:55.338549 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.338105 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52" event={"ID":"8ba6d31f-770c-4907-84e9-ce759ff12719","Type":"ContainerDied","Data":"a3d94915ac3cebdee712f93ed4f081f4c1499aafe92fe6f67892a455a1d69c22"} Nov 02 17:53:55.338549 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.338281 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3d94915ac3cebdee712f93ed4f081f4c1499aafe92fe6f67892a455a1d69c22" Nov 02 17:53:55.339579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.339556 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" Nov 02 17:53:55.339688 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.339595 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6" event={"ID":"6e642a3b-628a-4120-8db7-77005882e0af","Type":"ContainerDied","Data":"f80fe7eb20f9bdda9f810eeb2987b288d58156db23facef452e357f63078eb6e"} Nov 02 17:53:55.339688 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.339635 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f80fe7eb20f9bdda9f810eeb2987b288d58156db23facef452e357f63078eb6e" Nov 02 17:53:55.367102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.342584 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e642a3b-628a-4120-8db7-77005882e0af-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:55.367102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.342620 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6lfc\" (UniqueName: \"kubernetes.io/projected/6e642a3b-628a-4120-8db7-77005882e0af-kube-api-access-t6lfc\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:55.508102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.506861 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7"] Nov 02 17:53:55.508102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.507932 4310 cpu_manager.go:414] "containerMap: removing container" podUID="8ba6d31f-770c-4907-84e9-ce759ff12719" containerName="revoke-keys" Nov 02 17:53:55.508102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.507954 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba6d31f-770c-4907-84e9-ce759ff12719" containerName="revoke-keys" Nov 02 17:53:55.508102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.508002 4310 cpu_manager.go:414] "containerMap: removing container" podUID="6e642a3b-628a-4120-8db7-77005882e0af" containerName="revoke-keys" Nov 02 17:53:55.508102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.508011 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e642a3b-628a-4120-8db7-77005882e0af" containerName="revoke-keys" Nov 02 17:53:55.508492 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.508242 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e642a3b-628a-4120-8db7-77005882e0af" containerName="revoke-keys" Nov 02 17:53:55.508492 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.508269 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba6d31f-770c-4907-84e9-ce759ff12719" containerName="revoke-keys" Nov 02 17:53:55.518360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.518143 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:55.518360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.518183 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7"] Nov 02 17:53:55.528002 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.527786 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:53:55.528002 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.527878 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:53:55.609292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.609254 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7427c3d9-xk7p7\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:55.609505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.609419 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7427c3d9-xk7p7\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:55.609505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.609456 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpxd8\" (UniqueName: \"kubernetes.io/projected/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-kube-api-access-jpxd8\") pod \"maas-api-revoke-sub-7427c3d9-xk7p7\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:55.711587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.711503 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7427c3d9-xk7p7\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:55.712020 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.712000 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7427c3d9-xk7p7\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:55.712262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.712241 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpxd8\" (UniqueName: \"kubernetes.io/projected/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-kube-api-access-jpxd8\") pod \"maas-api-revoke-sub-7427c3d9-xk7p7\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:55.712951 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.712925 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7427c3d9-xk7p7\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:55.714728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.714708 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7427c3d9-xk7p7\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:55.808753 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.750962 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpxd8\" (UniqueName: \"kubernetes.io/projected/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-kube-api-access-jpxd8\") pod \"maas-api-revoke-sub-7427c3d9-xk7p7\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:55.808753 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.762876 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:55.895101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:55.883394 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:56.096103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:56.095540 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:53:56.096103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:56.095606 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 17:53:56.235343 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:56.235312 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:53:56.235565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:56.235540 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 17:53:56.401639 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:56.401122 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7"] Nov 02 17:53:56.904230 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:56.904203 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:57.049535 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:57.049098 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:53:57.049535 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:57.049160 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 17:53:57.236479 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:57.236439 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:53:57.236664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:57.236488 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 17:53:57.425580 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:57.425477 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" event={"ID":"1acfc3a9-75e8-461d-a29b-69eb0a4167f6","Type":"ContainerStarted","Data":"dcae7957dcc1cf3630fa2df988b7853347f07f6dc54cd4e63864673803eeaaf1"} Nov 02 17:53:57.426143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:57.426120 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" event={"ID":"1acfc3a9-75e8-461d-a29b-69eb0a4167f6","Type":"ContainerStarted","Data":"c07322b277b09568a1610c49d6855b4f5eac04fa1ea5e52cf7402498d729208e"} Nov 02 17:53:58.049239 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:58.046452 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:53:58.049239 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:58.046515 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 17:53:58.070102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:58.069483 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:58.244102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:58.239473 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:53:58.244102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:58.239538 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 17:53:58.436038 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:58.435955 4310 generic.go:356] "Generic (PLEG): container finished" podID="1acfc3a9-75e8-461d-a29b-69eb0a4167f6" containerID="dcae7957dcc1cf3630fa2df988b7853347f07f6dc54cd4e63864673803eeaaf1" exitCode=0 Nov 02 17:53:58.436476 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:58.436030 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" event={"ID":"1acfc3a9-75e8-461d-a29b-69eb0a4167f6","Type":"ContainerDied","Data":"dcae7957dcc1cf3630fa2df988b7853347f07f6dc54cd4e63864673803eeaaf1"} Nov 02 17:53:59.066103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.061479 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:53:59.066103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.061548 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 17:53:59.141897 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.141857 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:53:59.248102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.245149 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:53:59.248102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.245216 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 17:53:59.839832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.839800 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:53:59.907009 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.906975 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-kube-api-access-jpxd8\" (UniqueName: \"kubernetes.io/projected/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-kube-api-access-jpxd8\") pod \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " Nov 02 17:53:59.907188 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.907169 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-openshift-ca\") pod \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " Nov 02 17:53:59.907266 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.907239 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-xks-ca\") pod \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\" (UID: \"1acfc3a9-75e8-461d-a29b-69eb0a4167f6\") " Nov 02 17:53:59.907746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.907713 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-openshift-ca" pod "1acfc3a9-75e8-461d-a29b-69eb0a4167f6" (UID: "1acfc3a9-75e8-461d-a29b-69eb0a4167f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:59.908006 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.907982 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:53:59.910371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.910344 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-kube-api-access-jpxd8" pod "1acfc3a9-75e8-461d-a29b-69eb0a4167f6" (UID: "1acfc3a9-75e8-461d-a29b-69eb0a4167f6"). InnerVolumeSpecName "kube-api-access-jpxd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:59.912247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:53:59.912220 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-xks-ca" pod "1acfc3a9-75e8-461d-a29b-69eb0a4167f6" (UID: "1acfc3a9-75e8-461d-a29b-69eb0a4167f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:00.009122 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:00.009066 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:00.009122 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:00.009129 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpxd8\" (UniqueName: \"kubernetes.io/projected/1acfc3a9-75e8-461d-a29b-69eb0a4167f6-kube-api-access-jpxd8\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:00.045153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:00.044303 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:54:00.045153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:00.044366 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 17:54:00.182249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:00.182167 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:00.248509 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:00.248470 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 17:54:00.248788 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:00.248758 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 17:54:00.514727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:00.514683 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" event={"ID":"1acfc3a9-75e8-461d-a29b-69eb0a4167f6","Type":"ContainerDied","Data":"c07322b277b09568a1610c49d6855b4f5eac04fa1ea5e52cf7402498d729208e"} Nov 02 17:54:00.514908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:00.514756 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c07322b277b09568a1610c49d6855b4f5eac04fa1ea5e52cf7402498d729208e" Nov 02 17:54:00.514908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:00.514704 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7" Nov 02 17:54:01.259889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.259831 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:01.527853 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.526668 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w"] Nov 02 17:54:01.529534 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.529495 4310 cpu_manager.go:414] "containerMap: removing container" podUID="1acfc3a9-75e8-461d-a29b-69eb0a4167f6" containerName="revoke-keys" Nov 02 17:54:01.529714 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.529696 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acfc3a9-75e8-461d-a29b-69eb0a4167f6" containerName="revoke-keys" Nov 02 17:54:01.530146 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.530126 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acfc3a9-75e8-461d-a29b-69eb0a4167f6" containerName="revoke-keys" Nov 02 17:54:01.549453 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.549422 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:01.558512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.558485 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:54:01.558756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.558738 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:54:01.573144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.572301 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w"] Nov 02 17:54:01.666927 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.666890 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5sp\" (UniqueName: \"kubernetes.io/projected/00a2dccb-8da1-4236-a84d-51f0319d1a8e-kube-api-access-ld5sp\") pod \"maas-api-revoke-sub-00aa2fff-dpp6w\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:01.667175 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.666976 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00aa2fff-dpp6w\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:01.667175 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.667156 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00aa2fff-dpp6w\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:01.738046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.738017 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:54:01.738602 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.738577 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:54:01.739577 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.739558 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:54:01.769347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.768960 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5sp\" (UniqueName: \"kubernetes.io/projected/00a2dccb-8da1-4236-a84d-51f0319d1a8e-kube-api-access-ld5sp\") pod \"maas-api-revoke-sub-00aa2fff-dpp6w\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:01.769508 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.769405 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00aa2fff-dpp6w\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:01.770250 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.770226 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00aa2fff-dpp6w\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:01.771021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.770996 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00aa2fff-dpp6w\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:01.772030 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.771978 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00aa2fff-dpp6w\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:01.778511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.778453 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5sp\" (UniqueName: \"kubernetes.io/projected/00a2dccb-8da1-4236-a84d-51f0319d1a8e-kube-api-access-ld5sp\") pod \"maas-api-revoke-sub-00aa2fff-dpp6w\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:01.836298 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.836267 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:54:01.837949 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.837919 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:54:01.838848 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.838813 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:54:01.884521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:01.884486 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:02.194401 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:02.194331 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:02.209411 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:02.209382 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a2dccb_8da1_4236_a84d_51f0319d1a8e.slice/crio-c05486b3c5c404f155b57b0f056748a71c93038c2318459e31d120d136f91718 WatchSource:0}: Error finding container c05486b3c5c404f155b57b0f056748a71c93038c2318459e31d120d136f91718: Status 404 returned error can't find the container with id c05486b3c5c404f155b57b0f056748a71c93038c2318459e31d120d136f91718 Nov 02 17:54:02.212524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:02.212499 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w"] Nov 02 17:54:02.575670 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:02.575635 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" event={"ID":"00a2dccb-8da1-4236-a84d-51f0319d1a8e","Type":"ContainerStarted","Data":"c05486b3c5c404f155b57b0f056748a71c93038c2318459e31d120d136f91718"} Nov 02 17:54:02.972404 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:02.972372 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:03.525428 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:54:03.525341 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:54:03.586930 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:03.586895 4310 generic.go:356] "Generic (PLEG): container finished" podID="00a2dccb-8da1-4236-a84d-51f0319d1a8e" containerID="964990334dfb586323be9a359f2be7a2e4e86191058cc5d678098a2f9067eec0" exitCode=0 Nov 02 17:54:03.587449 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:03.586988 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" event={"ID":"00a2dccb-8da1-4236-a84d-51f0319d1a8e","Type":"ContainerDied","Data":"964990334dfb586323be9a359f2be7a2e4e86191058cc5d678098a2f9067eec0"} Nov 02 17:54:03.708539 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:03.708498 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:04.378898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:04.378865 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:04.778588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:04.778560 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:04.855529 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:04.855493 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-xks-ca\") pod \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " Nov 02 17:54:04.855707 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:04.855573 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a2dccb-8da1-4236-a84d-51f0319d1a8e-kube-api-access-ld5sp\" (UniqueName: \"kubernetes.io/projected/00a2dccb-8da1-4236-a84d-51f0319d1a8e-kube-api-access-ld5sp\") pod \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " Nov 02 17:54:04.855869 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:04.855848 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-openshift-ca\") pod \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\" (UID: \"00a2dccb-8da1-4236-a84d-51f0319d1a8e\") " Nov 02 17:54:04.856302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:04.856270 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-openshift-ca" pod "00a2dccb-8da1-4236-a84d-51f0319d1a8e" (UID: "00a2dccb-8da1-4236-a84d-51f0319d1a8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:04.856711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:04.856682 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:04.857900 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:04.857876 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a2dccb-8da1-4236-a84d-51f0319d1a8e-kube-api-access-ld5sp" pod "00a2dccb-8da1-4236-a84d-51f0319d1a8e" (UID: "00a2dccb-8da1-4236-a84d-51f0319d1a8e"). InnerVolumeSpecName "kube-api-access-ld5sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:04.858228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:04.858202 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-xks-ca" pod "00a2dccb-8da1-4236-a84d-51f0319d1a8e" (UID: "00a2dccb-8da1-4236-a84d-51f0319d1a8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:04.958473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:04.958396 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a2dccb-8da1-4236-a84d-51f0319d1a8e-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:04.958473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:04.958426 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld5sp\" (UniqueName: \"kubernetes.io/projected/00a2dccb-8da1-4236-a84d-51f0319d1a8e-kube-api-access-ld5sp\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:05.064601 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:05.064567 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:05.604600 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:05.604569 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" Nov 02 17:54:05.604600 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:05.604587 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w" event={"ID":"00a2dccb-8da1-4236-a84d-51f0319d1a8e","Type":"ContainerDied","Data":"c05486b3c5c404f155b57b0f056748a71c93038c2318459e31d120d136f91718"} Nov 02 17:54:05.604839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:05.604669 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c05486b3c5c404f155b57b0f056748a71c93038c2318459e31d120d136f91718" Nov 02 17:54:05.775629 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:05.775599 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:06.454062 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:06.454011 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:07.438103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.428893 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:07.938737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.938703 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd"] Nov 02 17:54:07.939659 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.939637 4310 cpu_manager.go:414] "containerMap: removing container" podUID="00a2dccb-8da1-4236-a84d-51f0319d1a8e" containerName="revoke-keys" Nov 02 17:54:07.939769 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.939660 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a2dccb-8da1-4236-a84d-51f0319d1a8e" containerName="revoke-keys" Nov 02 17:54:07.939859 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.939845 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a2dccb-8da1-4236-a84d-51f0319d1a8e" containerName="revoke-keys" Nov 02 17:54:07.943905 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.943882 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:07.947873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.947836 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:54:07.948685 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.948666 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:54:07.953117 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.953091 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd"] Nov 02 17:54:07.978228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.978194 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br"] Nov 02 17:54:07.984887 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.984850 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:07.990747 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:07.990717 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br"] Nov 02 17:54:08.024352 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.024318 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5ea585-bbkkd\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:08.024526 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.024365 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5ea585-bbkkd\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:08.024526 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.024477 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9mk7\" (UniqueName: \"kubernetes.io/projected/abb3cc13-1c8c-4183-ab6a-d2310b054d45-kube-api-access-z9mk7\") pod \"maas-api-revoke-sub-ca5ea585-bbkkd\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:08.126107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.126039 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntdp\" (UniqueName: \"kubernetes.io/projected/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-kube-api-access-cntdp\") pod \"maas-api-revoke-sub-63487414-4d7br\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:08.126294 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.126198 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5ea585-bbkkd\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:08.126294 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.126239 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5ea585-bbkkd\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:08.126294 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.126280 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9mk7\" (UniqueName: \"kubernetes.io/projected/abb3cc13-1c8c-4183-ab6a-d2310b054d45-kube-api-access-z9mk7\") pod \"maas-api-revoke-sub-ca5ea585-bbkkd\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:08.126526 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.126507 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63487414-4d7br\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:08.126733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.126712 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63487414-4d7br\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:08.126833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.126816 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5ea585-bbkkd\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:08.128685 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.128665 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5ea585-bbkkd\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:08.135785 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.135760 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9mk7\" (UniqueName: \"kubernetes.io/projected/abb3cc13-1c8c-4183-ab6a-d2310b054d45-kube-api-access-z9mk7\") pod \"maas-api-revoke-sub-ca5ea585-bbkkd\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:08.228722 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.228688 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63487414-4d7br\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:08.228914 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.228741 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntdp\" (UniqueName: \"kubernetes.io/projected/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-kube-api-access-cntdp\") pod \"maas-api-revoke-sub-63487414-4d7br\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:08.228989 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.228916 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63487414-4d7br\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:08.229421 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.229395 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63487414-4d7br\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:08.231263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.231243 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63487414-4d7br\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:08.243121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.240786 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntdp\" (UniqueName: \"kubernetes.io/projected/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-kube-api-access-cntdp\") pod \"maas-api-revoke-sub-63487414-4d7br\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:08.264024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.263990 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:08.302883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.302856 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:08.386193 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.386160 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:08.536154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.532313 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5"] Nov 02 17:54:08.551986 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.542676 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:08.578149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.570065 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5"] Nov 02 17:54:08.602731 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.602701 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd"] Nov 02 17:54:08.624313 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.623696 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725ef6d4-4h4z5\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:08.624313 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.623992 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725ef6d4-4h4z5\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:08.624313 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.624103 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kgmb\" (UniqueName: \"kubernetes.io/projected/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-kube-api-access-5kgmb\") pod \"maas-api-revoke-sub-725ef6d4-4h4z5\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:08.628561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.628519 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br"] Nov 02 17:54:08.638654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.638624 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" event={"ID":"abb3cc13-1c8c-4183-ab6a-d2310b054d45","Type":"ContainerStarted","Data":"cf8d17d80c531fb02d34d8b5ec25e0fe0d680ab5a1cb9088fb4968c64bf3e1a3"} Nov 02 17:54:08.725174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.725128 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725ef6d4-4h4z5\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:08.725373 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.725232 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kgmb\" (UniqueName: \"kubernetes.io/projected/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-kube-api-access-5kgmb\") pod \"maas-api-revoke-sub-725ef6d4-4h4z5\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:08.725373 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.725313 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725ef6d4-4h4z5\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:08.726013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.725992 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725ef6d4-4h4z5\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:08.727654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.727635 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725ef6d4-4h4z5\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:08.733205 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.733178 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kgmb\" (UniqueName: \"kubernetes.io/projected/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-kube-api-access-5kgmb\") pod \"maas-api-revoke-sub-725ef6d4-4h4z5\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:08.786710 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.786628 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn"] Nov 02 17:54:08.794766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.794741 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:08.798265 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.798227 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn"] Nov 02 17:54:08.827042 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.826992 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbaaebc6-5smcn\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:08.827313 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.827286 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbaaebc6-5smcn\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:08.827495 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.827470 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9r8\" (UniqueName: \"kubernetes.io/projected/208e4394-235b-431d-9e76-a06f2feb9f28-kube-api-access-ds9r8\") pod \"maas-api-revoke-sub-fbaaebc6-5smcn\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:08.899777 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.899739 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:08.929227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.929194 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbaaebc6-5smcn\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:08.929350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.929248 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbaaebc6-5smcn\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:08.929350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.929321 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9r8\" (UniqueName: \"kubernetes.io/projected/208e4394-235b-431d-9e76-a06f2feb9f28-kube-api-access-ds9r8\") pod \"maas-api-revoke-sub-fbaaebc6-5smcn\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:08.929921 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.929900 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbaaebc6-5smcn\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:08.931534 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.931515 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbaaebc6-5smcn\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:08.937742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:08.937721 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9r8\" (UniqueName: \"kubernetes.io/projected/208e4394-235b-431d-9e76-a06f2feb9f28-kube-api-access-ds9r8\") pod \"maas-api-revoke-sub-fbaaebc6-5smcn\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:09.122447 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.122410 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:09.274977 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:09.274553 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e577ac_e4cf_4b98_8243_332c9be8c7f3.slice/crio-a88aeff6da5e822e5c9e0e626bf4e189c393cc253c131adddb6130b70a25a1c3 WatchSource:0}: Error finding container a88aeff6da5e822e5c9e0e626bf4e189c393cc253c131adddb6130b70a25a1c3: Status 404 returned error can't find the container with id a88aeff6da5e822e5c9e0e626bf4e189c393cc253c131adddb6130b70a25a1c3 Nov 02 17:54:09.278134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.278104 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5"] Nov 02 17:54:09.278518 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.278373 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:09.467477 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.467456 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn"] Nov 02 17:54:09.470610 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:09.470584 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208e4394_235b_431d_9e76_a06f2feb9f28.slice/crio-3097e3ae71951ca093dc9ef1a6b35ac0314f94b0ce032a4753f98901ec71a30a WatchSource:0}: Error finding container 3097e3ae71951ca093dc9ef1a6b35ac0314f94b0ce032a4753f98901ec71a30a: Status 404 returned error can't find the container with id 3097e3ae71951ca093dc9ef1a6b35ac0314f94b0ce032a4753f98901ec71a30a Nov 02 17:54:09.651999 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.651959 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" event={"ID":"a4e577ac-e4cf-4b98-8243-332c9be8c7f3","Type":"ContainerStarted","Data":"a88aeff6da5e822e5c9e0e626bf4e189c393cc253c131adddb6130b70a25a1c3"} Nov 02 17:54:09.653530 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.653502 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" event={"ID":"208e4394-235b-431d-9e76-a06f2feb9f28","Type":"ContainerStarted","Data":"3097e3ae71951ca093dc9ef1a6b35ac0314f94b0ce032a4753f98901ec71a30a"} Nov 02 17:54:09.655534 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.655510 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" event={"ID":"d4e49a42-496d-4efe-87ac-70a9bfa43dfc","Type":"ContainerStarted","Data":"98d448aff8b4395afb8b83fc5769c44117acd32c147822ed163ff9a3e6ebecae"} Nov 02 17:54:09.655633 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.655554 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" event={"ID":"d4e49a42-496d-4efe-87ac-70a9bfa43dfc","Type":"ContainerStarted","Data":"fef1d9507c784dee8bc6dd8c60c0984a648263ebcb05b64239e1d293a98e13e7"} Nov 02 17:54:09.657496 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.657469 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" event={"ID":"abb3cc13-1c8c-4183-ab6a-d2310b054d45","Type":"ContainerStarted","Data":"be233ef3a371e4d01f0ccc99f8d2b014d132ede8e369b97705c5c5d1a9361750"} Nov 02 17:54:09.675169 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.675120 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" podStartSLOduration=2.675067834 podStartE2EDuration="2.675067834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:09.673642806 +0000 UTC m=+847.116725443" watchObservedRunningTime="2026-11-02 17:54:09.675067834 +0000 UTC m=+847.118150470" Nov 02 17:54:09.693436 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.693392 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" podStartSLOduration=2.6933800310000002 podStartE2EDuration="2.693380031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:09.692882071 +0000 UTC m=+847.135964713" watchObservedRunningTime="2026-11-02 17:54:09.693380031 +0000 UTC m=+847.136462668" Nov 02 17:54:09.978780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:09.978744 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:11.058059 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.058015 4310 generic.go:356] "Generic (PLEG): container finished" podID="abb3cc13-1c8c-4183-ab6a-d2310b054d45" containerID="be233ef3a371e4d01f0ccc99f8d2b014d132ede8e369b97705c5c5d1a9361750" exitCode=0 Nov 02 17:54:11.103111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.085863 4310 generic.go:356] "Generic (PLEG): container finished" podID="a4e577ac-e4cf-4b98-8243-332c9be8c7f3" containerID="855b95009d09dec8f3e7b549b3b581c02249bc99172be0fa844c7cab672e3437" exitCode=0 Nov 02 17:54:11.129831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.129675 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" event={"ID":"abb3cc13-1c8c-4183-ab6a-d2310b054d45","Type":"ContainerDied","Data":"be233ef3a371e4d01f0ccc99f8d2b014d132ede8e369b97705c5c5d1a9361750"} Nov 02 17:54:11.129831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.129772 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" event={"ID":"a4e577ac-e4cf-4b98-8243-332c9be8c7f3","Type":"ContainerDied","Data":"855b95009d09dec8f3e7b549b3b581c02249bc99172be0fa844c7cab672e3437"} Nov 02 17:54:11.129831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.129819 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" event={"ID":"208e4394-235b-431d-9e76-a06f2feb9f28","Type":"ContainerStarted","Data":"80ba3f95e1d4a2b57830444628520e361087f7f72d56be881c301104dfdc83c0"} Nov 02 17:54:11.252636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.252586 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:54:11.328836 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.328746 4310 generic.go:356] "Generic (PLEG): container finished" podID="d4e49a42-496d-4efe-87ac-70a9bfa43dfc" containerID="98d448aff8b4395afb8b83fc5769c44117acd32c147822ed163ff9a3e6ebecae" exitCode=0 Nov 02 17:54:11.329168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.329143 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" event={"ID":"d4e49a42-496d-4efe-87ac-70a9bfa43dfc","Type":"ContainerDied","Data":"98d448aff8b4395afb8b83fc5769c44117acd32c147822ed163ff9a3e6ebecae"} Nov 02 17:54:11.735100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.723345 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:11.735100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.723783 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:11.822149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.819506 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:11.822149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:11.821899 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.018866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:12.018756 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:54:12.021692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:12.019641 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:54:12.021692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:12.020572 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.130235 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:12.130164 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Nov 02 17:54:12.279157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:12.276060 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.279157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:12.276553 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:54:12.749145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:12.723642 4310 generic.go:356] "Generic (PLEG): container finished" podID="208e4394-235b-431d-9e76-a06f2feb9f28" containerID="80ba3f95e1d4a2b57830444628520e361087f7f72d56be881c301104dfdc83c0" exitCode=0 Nov 02 17:54:12.751617 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:12.751577 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:54:12.751776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:12.751755 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:54:12.844114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:12.831219 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:12.967867 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:12.876096 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:54:13.126670 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.126381 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.129788 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.129753 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.216166 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.137804 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:54:13.283998 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.282370 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.352632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.352594 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:13.412701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.412599 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:54:13.412881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.412737 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:54:13.412881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.412758 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" event={"ID":"208e4394-235b-431d-9e76-a06f2feb9f28","Type":"ContainerDied","Data":"80ba3f95e1d4a2b57830444628520e361087f7f72d56be881c301104dfdc83c0"} Nov 02 17:54:13.412881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.412816 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:54:13.412881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.412854 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:54:13.413135 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.412888 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:54:13.413135 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.412920 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:54:13.413135 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.412956 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:54:13.422437 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:13.422243 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded" Nov 02 17:54:14.035251 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:14.035207 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:14.061106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:14.051955 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:14.136103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:14.133017 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:14.137281 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:14.137226 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:14.279143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:14.279093 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:54:14.315207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:14.310793 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:14.331245 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:14.330190 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:14.331245 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:14.330489 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:54:14.387165 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:14.359850 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:14.653297 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:14.647009 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:54:14.653700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:14.653474 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:15.123970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:15.105260 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:54:15.123970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:15.122362 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:15.576183 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:15.568264 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:15.638104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:15.634753 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:54:15.813130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:15.811136 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" podUID="c19cdb66-3f5e-4ac8-a77c-d0a85d130613" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:15.950676 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:15.950560 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:54:15.980110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:15.959549 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:54:15.981857 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:15.981818 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:54:16.024024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:16.023977 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.410550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:16.410512 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:54:16.568444 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:16.562370 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:16.593714 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:16.593670 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:54:16.707589 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:16.707483 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.710466 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:16.710435 4310 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-bvzg2 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.7:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:16.710682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:16.710654 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bvzg2" podUID="87d8bb05-337f-4eb5-b71f-f95f3b160ecb" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.133.0.7:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:16.759365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:16.742533 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": context deadline exceeded" Nov 02 17:54:16.759365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:16.742634 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:16.986575 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:16.972765 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:17.180102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.162071 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:17.180102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.162182 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.192238 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:54:17.192201 4310 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.487s" Nov 02 17:54:17.192512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.192494 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:54:17.240057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.239954 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.362266 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.357448 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.362266 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.253235 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:54:17.362543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.253160 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:54:17.376111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.365289 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:54:17.376111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:16.952319 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" podUID="c19cdb66-3f5e-4ac8-a77c-d0a85d130613" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Nov 02 17:54:17.376414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.287450 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" podUID="378c5f34-0627-4df2-8096-5bb1e582c650" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 02 17:54:17.398443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.287152 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:17.398748 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.398705 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.398898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.372204 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:17.399036 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.399015 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.400010 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.399981 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:17.487116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.452266 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.487116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.417211 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.487116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.458708 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:17.487116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.458773 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.487116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.468235 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:17.487116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.468295 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.542279 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.540743 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:54:17.648105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.632022 4310 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.824759634s: [/var/lib/containers/storage/overlay/94c73ed40c9d394504a663395b6843d826e08a7122e55aebbcecbee01262bc84/diff /var/log/pods/openshift-insights_insights-runtime-extractor-zg6hp_5350eca9-94c7-4dc9-a571-ec16e055d94d/extractor/0.log]; will not log again for this container unless duration exceeds 2s Nov 02 17:54:17.682291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.682238 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" podUID="af852c12-fc47-48e9-ac51-7418a37792d5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.53:50052\" within 1s: context deadline exceeded" Nov 02 17:54:17.705068 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.691146 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:17.705068 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.691218 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.722671 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.722626 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:54:17.934535 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.934407 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:17.935217 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.723491 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 02 17:54:17.935408 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:17.757872 4310 logging.go:55] [core] [Channel #799 SubChannel #801]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.53:50052", ServerName: "10.133.0.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.53:50052: operation was canceled" Nov 02 17:54:17.935504 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.924194 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:17.935623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.935598 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:17.935752 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.831355 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:18.028444 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:17.723461 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:54:18.109252 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.104499 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" Nov 02 17:54:18.249103 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:54:18.232548 4310 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.04s" Nov 02 17:54:18.249103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.232592 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5" event={"ID":"a4e577ac-e4cf-4b98-8243-332c9be8c7f3","Type":"ContainerDied","Data":"a88aeff6da5e822e5c9e0e626bf4e189c393cc253c131adddb6130b70a25a1c3"} Nov 02 17:54:18.249103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.232648 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a88aeff6da5e822e5c9e0e626bf4e189c393cc253c131adddb6130b70a25a1c3" Nov 02 17:54:18.249103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.232725 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:54:18.249103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.232855 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:54:18.249103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.232901 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.370893 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.456219 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.482585 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.482677 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.525164 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.526547 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.526596 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.526679 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.526703 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.526745 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.526765 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.526813 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:18.528160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.526836 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:18.541372 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.541328 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:54:18.563604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.555201 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:54:18.563604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.555571 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:54:18.564455 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.564409 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:18.564693 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.564662 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:18.597110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.590741 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-kube-api-access-5kgmb\" (UniqueName: \"kubernetes.io/projected/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-kube-api-access-5kgmb\") pod \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " Nov 02 17:54:18.597110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.590838 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208e4394-235b-431d-9e76-a06f2feb9f28-kube-api-access-ds9r8\" (UniqueName: \"kubernetes.io/projected/208e4394-235b-431d-9e76-a06f2feb9f28-kube-api-access-ds9r8\") pod \"208e4394-235b-431d-9e76-a06f2feb9f28\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " Nov 02 17:54:18.597110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.590881 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-openshift-ca\") pod \"208e4394-235b-431d-9e76-a06f2feb9f28\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " Nov 02 17:54:18.597110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.590928 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-openshift-ca\") pod \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " Nov 02 17:54:18.597110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.591021 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-xks-ca\") pod \"208e4394-235b-431d-9e76-a06f2feb9f28\" (UID: \"208e4394-235b-431d-9e76-a06f2feb9f28\") " Nov 02 17:54:18.597110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.591159 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-xks-ca\") pod \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " Nov 02 17:54:18.603101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.597750 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-openshift-ca" pod "208e4394-235b-431d-9e76-a06f2feb9f28" (UID: "208e4394-235b-431d-9e76-a06f2feb9f28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:18.638440 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.638412 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:18.707162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.694335 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-openshift-ca" pod "a4e577ac-e4cf-4b98-8243-332c9be8c7f3" (UID: "a4e577ac-e4cf-4b98-8243-332c9be8c7f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:18.958187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.936762 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:18.958584 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.958545 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-kube-api-access-5kgmb" pod "a4e577ac-e4cf-4b98-8243-332c9be8c7f3" (UID: "a4e577ac-e4cf-4b98-8243-332c9be8c7f3"). InnerVolumeSpecName "kube-api-access-5kgmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:19.234684 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:18.995188 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:19.234684 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.046992 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kgmb\" (UniqueName: \"kubernetes.io/projected/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-kube-api-access-5kgmb\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:19.234684 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.047027 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:19.234684 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.071184 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:19.315016 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.236233 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:19.315016 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.236329 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded" Nov 02 17:54:19.406107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.403424 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:19.461669 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.456648 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:19.565194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.472397 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-xks-ca" pod "208e4394-235b-431d-9e76-a06f2feb9f28" (UID: "208e4394-235b-431d-9e76-a06f2feb9f28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:19.565194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.477886 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208e4394-235b-431d-9e76-a06f2feb9f28-kube-api-access-ds9r8" pod "208e4394-235b-431d-9e76-a06f2feb9f28" (UID: "208e4394-235b-431d-9e76-a06f2feb9f28"). InnerVolumeSpecName "kube-api-access-ds9r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:19.565194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.562780 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:19.565194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.561173 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:19.588309 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.588267 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" Nov 02 17:54:19.616259 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.563734 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:19.617588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.563770 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:54:19.733159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.698792 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" podUID="3b41c5ba-f619-4894-8e4f-427014637518" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Nov 02 17:54:19.733159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.714158 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-xks-ca" pod "a4e577ac-e4cf-4b98-8243-332c9be8c7f3" (UID: "a4e577ac-e4cf-4b98-8243-332c9be8c7f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:19.902740 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.902276 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:19.927102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.910291 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-xks-ca\") pod \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\" (UID: \"a4e577ac-e4cf-4b98-8243-332c9be8c7f3\") " Nov 02 17:54:19.927102 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:19.910429 4310 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/a4e577ac-e4cf-4b98-8243-332c9be8c7f3/volumes/kubernetes.io~secret/maas-api-xks-ca Nov 02 17:54:19.927102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.925174 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-xks-ca" pod "a4e577ac-e4cf-4b98-8243-332c9be8c7f3" (UID: "a4e577ac-e4cf-4b98-8243-332c9be8c7f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:19.973789 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.972139 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:19.993907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.993775 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e577ac-e4cf-4b98-8243-332c9be8c7f3-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:19.994272 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.994251 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds9r8\" (UniqueName: \"kubernetes.io/projected/208e4394-235b-431d-9e76-a06f2feb9f28-kube-api-access-ds9r8\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:19.994428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.994414 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208e4394-235b-431d-9e76-a06f2feb9f28-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:20.018588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:19.997255 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:20.176156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.072335 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:20.176156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.072699 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded" Nov 02 17:54:20.271159 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:54:20.109056 4310 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.434s" Nov 02 17:54:20.271651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.271615 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn" event={"ID":"208e4394-235b-431d-9e76-a06f2feb9f28","Type":"ContainerDied","Data":"3097e3ae71951ca093dc9ef1a6b35ac0314f94b0ce032a4753f98901ec71a30a"} Nov 02 17:54:20.317210 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.317168 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3097e3ae71951ca093dc9ef1a6b35ac0314f94b0ce032a4753f98901ec71a30a" Nov 02 17:54:20.357269 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.303266 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:20.482757 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.276759 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-xks-ca\") pod \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " Nov 02 17:54:20.482757 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.461354 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-openshift-ca\") pod \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " Nov 02 17:54:20.482757 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.461657 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb3cc13-1c8c-4183-ab6a-d2310b054d45-kube-api-access-z9mk7\" (UniqueName: \"kubernetes.io/projected/abb3cc13-1c8c-4183-ab6a-d2310b054d45-kube-api-access-z9mk7\") pod \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\" (UID: \"abb3cc13-1c8c-4183-ab6a-d2310b054d45\") " Nov 02 17:54:20.482757 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.467193 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-openshift-ca" pod "abb3cc13-1c8c-4183-ab6a-d2310b054d45" (UID: "abb3cc13-1c8c-4183-ab6a-d2310b054d45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:20.567356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.566378 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:54:20.607449 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.605126 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:20.650196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.646543 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:54:20.742609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.739816 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:20.742609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.741640 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:20.745360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.744892 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-openshift-ca\") pod \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " Nov 02 17:54:20.745360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.744970 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-kube-api-access-cntdp\" (UniqueName: \"kubernetes.io/projected/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-kube-api-access-cntdp\") pod \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " Nov 02 17:54:20.745360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.745157 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-xks-ca\") pod \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\" (UID: \"d4e49a42-496d-4efe-87ac-70a9bfa43dfc\") " Nov 02 17:54:20.746864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.746834 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:20.747027 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.747001 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:20.751185 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.751160 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-xks-ca" pod "abb3cc13-1c8c-4183-ab6a-d2310b054d45" (UID: "abb3cc13-1c8c-4183-ab6a-d2310b054d45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:20.761299 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.761269 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-openshift-ca" pod "d4e49a42-496d-4efe-87ac-70a9bfa43dfc" (UID: "d4e49a42-496d-4efe-87ac-70a9bfa43dfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:20.779954 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.779921 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb3cc13-1c8c-4183-ab6a-d2310b054d45-kube-api-access-z9mk7" pod "abb3cc13-1c8c-4183-ab6a-d2310b054d45" (UID: "abb3cc13-1c8c-4183-ab6a-d2310b054d45"). InnerVolumeSpecName "kube-api-access-z9mk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:20.784368 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.784337 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-xks-ca" pod "d4e49a42-496d-4efe-87ac-70a9bfa43dfc" (UID: "d4e49a42-496d-4efe-87ac-70a9bfa43dfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:20.791652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.789232 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-kube-api-access-cntdp" pod "d4e49a42-496d-4efe-87ac-70a9bfa43dfc" (UID: "d4e49a42-496d-4efe-87ac-70a9bfa43dfc"). InnerVolumeSpecName "kube-api-access-cntdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:20.797167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.797145 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:54:20.801316 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.801297 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:54:20.808609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.807580 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:54:20.815299 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.815274 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:54:20.821829 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.821808 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:54:20.822035 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.822021 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:54:20.822986 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.822958 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:54:20.823136 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.823121 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:54:20.823801 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.823777 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" event={"ID":"d4e49a42-496d-4efe-87ac-70a9bfa43dfc","Type":"ContainerDied","Data":"fef1d9507c784dee8bc6dd8c60c0984a648263ebcb05b64239e1d293a98e13e7"} Nov 02 17:54:20.823895 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.823822 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fef1d9507c784dee8bc6dd8c60c0984a648263ebcb05b64239e1d293a98e13e7" Nov 02 17:54:20.823895 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.823859 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:54:20.824430 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.824410 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br" Nov 02 17:54:20.827381 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.827364 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" Nov 02 17:54:20.827576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.827457 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd" event={"ID":"abb3cc13-1c8c-4183-ab6a-d2310b054d45","Type":"ContainerDied","Data":"cf8d17d80c531fb02d34d8b5ec25e0fe0d680ab5a1cb9088fb4968c64bf3e1a3"} Nov 02 17:54:20.827735 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.827722 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf8d17d80c531fb02d34d8b5ec25e0fe0d680ab5a1cb9088fb4968c64bf3e1a3" Nov 02 17:54:20.849048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.848982 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9mk7\" (UniqueName: \"kubernetes.io/projected/abb3cc13-1c8c-4183-ab6a-d2310b054d45-kube-api-access-z9mk7\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:20.851843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.851820 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb3cc13-1c8c-4183-ab6a-d2310b054d45-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:20.852124 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.852105 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:20.852312 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.852293 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cntdp\" (UniqueName: \"kubernetes.io/projected/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-kube-api-access-cntdp\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:20.852442 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:20.852427 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e49a42-496d-4efe-87ac-70a9bfa43dfc-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:20.869851 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:54:20.869804 4310 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e577ac_e4cf_4b98_8243_332c9be8c7f3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e49a42_496d_4efe_87ac_70a9bfa43dfc.slice/crio-fef1d9507c784dee8bc6dd8c60c0984a648263ebcb05b64239e1d293a98e13e7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e49a42_496d_4efe_87ac_70a9bfa43dfc.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:54:21.389872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:21.389522 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:22.199673 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:22.199639 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:22.979957 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:22.979924 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:23.168100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.167215 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r"] Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171592 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a4e577ac-e4cf-4b98-8243-332c9be8c7f3" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171622 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e577ac-e4cf-4b98-8243-332c9be8c7f3" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171645 4310 cpu_manager.go:414] "containerMap: removing container" podUID="208e4394-235b-431d-9e76-a06f2feb9f28" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171654 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208e4394-235b-431d-9e76-a06f2feb9f28" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171676 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d4e49a42-496d-4efe-87ac-70a9bfa43dfc" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171685 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e49a42-496d-4efe-87ac-70a9bfa43dfc" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171748 4310 cpu_manager.go:414] "containerMap: removing container" podUID="abb3cc13-1c8c-4183-ab6a-d2310b054d45" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171755 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb3cc13-1c8c-4183-ab6a-d2310b054d45" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171949 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb3cc13-1c8c-4183-ab6a-d2310b054d45" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171966 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e577ac-e4cf-4b98-8243-332c9be8c7f3" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171976 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="208e4394-235b-431d-9e76-a06f2feb9f28" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.171986 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e49a42-496d-4efe-87ac-70a9bfa43dfc" containerName="revoke-keys" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.180701 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:23.182590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.182027 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r"] Nov 02 17:54:23.187212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.187181 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:54:23.188235 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.188196 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:54:23.236851 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.236767 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfx8c\" (UniqueName: \"kubernetes.io/projected/770cd47b-7a5d-4885-bae2-98948b2ddedd-kube-api-access-kfx8c\") pod \"maas-api-revoke-sub-f0782c3a-sgn8r\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:23.237355 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.236992 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0782c3a-sgn8r\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:23.237355 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.237047 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0782c3a-sgn8r\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:23.338503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.338470 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0782c3a-sgn8r\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:23.338652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.338515 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0782c3a-sgn8r\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:23.338765 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.338747 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfx8c\" (UniqueName: \"kubernetes.io/projected/770cd47b-7a5d-4885-bae2-98948b2ddedd-kube-api-access-kfx8c\") pod \"maas-api-revoke-sub-f0782c3a-sgn8r\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:23.339181 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.339158 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0782c3a-sgn8r\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:23.340980 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.340962 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0782c3a-sgn8r\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:23.346744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.346724 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfx8c\" (UniqueName: \"kubernetes.io/projected/770cd47b-7a5d-4885-bae2-98948b2ddedd-kube-api-access-kfx8c\") pod \"maas-api-revoke-sub-f0782c3a-sgn8r\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:23.502065 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.501983 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:23.790186 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:23.790156 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770cd47b_7a5d_4885_bae2_98948b2ddedd.slice/crio-46669bf1a421913e79cb71e52e060f46629f94e64b6f7ee0d1eddb493baa38ba WatchSource:0}: Error finding container 46669bf1a421913e79cb71e52e060f46629f94e64b6f7ee0d1eddb493baa38ba: Status 404 returned error can't find the container with id 46669bf1a421913e79cb71e52e060f46629f94e64b6f7ee0d1eddb493baa38ba Nov 02 17:54:23.793699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.793675 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r"] Nov 02 17:54:23.855873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.855843 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" event={"ID":"770cd47b-7a5d-4885-bae2-98948b2ddedd","Type":"ContainerStarted","Data":"46669bf1a421913e79cb71e52e060f46629f94e64b6f7ee0d1eddb493baa38ba"} Nov 02 17:54:23.886276 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:23.886241 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:24.331627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.331552 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw"] Nov 02 17:54:24.337298 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.336313 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:24.368040 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.368010 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw"] Nov 02 17:54:24.473070 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.473033 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a6f6ed-5mqzw\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:24.473261 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.473240 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a6f6ed-5mqzw\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:24.473308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.473276 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlm4\" (UniqueName: \"kubernetes.io/projected/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-kube-api-access-xnlm4\") pod \"maas-api-revoke-sub-03a6f6ed-5mqzw\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:24.574932 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.574893 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a6f6ed-5mqzw\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:24.574932 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.574952 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlm4\" (UniqueName: \"kubernetes.io/projected/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-kube-api-access-xnlm4\") pod \"maas-api-revoke-sub-03a6f6ed-5mqzw\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:24.575249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.575173 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a6f6ed-5mqzw\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:24.575644 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.575617 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a6f6ed-5mqzw\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:24.577539 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.577517 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a6f6ed-5mqzw\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:24.583966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.583905 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlm4\" (UniqueName: \"kubernetes.io/projected/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-kube-api-access-xnlm4\") pod \"maas-api-revoke-sub-03a6f6ed-5mqzw\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:24.678734 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.678703 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:24.939739 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.939563 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:24.992606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.992568 4310 generic.go:356] "Generic (PLEG): container finished" podID="770cd47b-7a5d-4885-bae2-98948b2ddedd" containerID="ad379257f8490a7f232b49f9e6b2fdc72046a4a28c8bf3177c2edcf0ef6f0ddf" exitCode=0 Nov 02 17:54:24.993025 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:24.992996 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" event={"ID":"770cd47b-7a5d-4885-bae2-98948b2ddedd","Type":"ContainerDied","Data":"ad379257f8490a7f232b49f9e6b2fdc72046a4a28c8bf3177c2edcf0ef6f0ddf"} Nov 02 17:54:25.004172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.004124 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z"] Nov 02 17:54:25.011445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.011325 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:25.013707 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.013641 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z"] Nov 02 17:54:25.026538 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.026510 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw"] Nov 02 17:54:25.134536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.134499 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbd13214-dn79z\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:25.134668 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.134545 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbd13214-dn79z\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:25.134908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.134894 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndbz8\" (UniqueName: \"kubernetes.io/projected/c3017323-41c0-4cde-9edb-ba9a6185351f-kube-api-access-ndbz8\") pod \"maas-api-revoke-sub-bbd13214-dn79z\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:25.236748 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.236711 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndbz8\" (UniqueName: \"kubernetes.io/projected/c3017323-41c0-4cde-9edb-ba9a6185351f-kube-api-access-ndbz8\") pod \"maas-api-revoke-sub-bbd13214-dn79z\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:25.236939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.236925 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbd13214-dn79z\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:25.236999 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.236955 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbd13214-dn79z\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:25.237647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.237625 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbd13214-dn79z\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:25.239500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.239474 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbd13214-dn79z\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:25.246325 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.246302 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndbz8\" (UniqueName: \"kubernetes.io/projected/c3017323-41c0-4cde-9edb-ba9a6185351f-kube-api-access-ndbz8\") pod \"maas-api-revoke-sub-bbd13214-dn79z\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:25.335060 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.335030 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:25.719563 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:25.719532 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3017323_41c0_4cde_9edb_ba9a6185351f.slice/crio-98c39a32a789dd6809dcd77e8d1c5229be9d7f2fd3af4fc81fe63a842385241a WatchSource:0}: Error finding container 98c39a32a789dd6809dcd77e8d1c5229be9d7f2fd3af4fc81fe63a842385241a: Status 404 returned error can't find the container with id 98c39a32a789dd6809dcd77e8d1c5229be9d7f2fd3af4fc81fe63a842385241a Nov 02 17:54:25.722783 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.722745 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z"] Nov 02 17:54:25.832630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:25.832581 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:26.001504 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.001431 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7"] Nov 02 17:54:26.022628 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.022360 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:26.050719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.050682 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7"] Nov 02 17:54:26.051788 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.051757 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18d8a44d-mn2h7\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:26.053198 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.052126 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18d8a44d-mn2h7\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:26.053198 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.052181 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlfvk\" (UniqueName: \"kubernetes.io/projected/4c519182-e8de-4e4a-8668-af65f7dea9b6-kube-api-access-rlfvk\") pod \"maas-api-revoke-sub-18d8a44d-mn2h7\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:26.071342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.071317 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" event={"ID":"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d","Type":"ContainerStarted","Data":"b9da58ac8049143892414e5578bcb4c56ec6e8bafe036d94466118dddd2fbd58"} Nov 02 17:54:26.071476 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.071465 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" event={"ID":"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d","Type":"ContainerStarted","Data":"80b782e5dc218d7822a886f24f188c4427daaf382bf3ee2899c5fb1c483c0691"} Nov 02 17:54:26.073030 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.073010 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" event={"ID":"c3017323-41c0-4cde-9edb-ba9a6185351f","Type":"ContainerStarted","Data":"98c39a32a789dd6809dcd77e8d1c5229be9d7f2fd3af4fc81fe63a842385241a"} Nov 02 17:54:26.154148 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.154114 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18d8a44d-mn2h7\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:26.154315 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.154162 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlfvk\" (UniqueName: \"kubernetes.io/projected/4c519182-e8de-4e4a-8668-af65f7dea9b6-kube-api-access-rlfvk\") pod \"maas-api-revoke-sub-18d8a44d-mn2h7\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:26.154315 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.154256 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18d8a44d-mn2h7\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:26.157546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.157514 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18d8a44d-mn2h7\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:26.159468 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.159435 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18d8a44d-mn2h7\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:26.164871 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.164845 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlfvk\" (UniqueName: \"kubernetes.io/projected/4c519182-e8de-4e4a-8668-af65f7dea9b6-kube-api-access-rlfvk\") pod \"maas-api-revoke-sub-18d8a44d-mn2h7\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:26.235469 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.235442 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:26.256878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.255771 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-openshift-ca\") pod \"770cd47b-7a5d-4885-bae2-98948b2ddedd\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " Nov 02 17:54:26.256878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.255840 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770cd47b-7a5d-4885-bae2-98948b2ddedd-kube-api-access-kfx8c\" (UniqueName: \"kubernetes.io/projected/770cd47b-7a5d-4885-bae2-98948b2ddedd-kube-api-access-kfx8c\") pod \"770cd47b-7a5d-4885-bae2-98948b2ddedd\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " Nov 02 17:54:26.256878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.255914 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-xks-ca\") pod \"770cd47b-7a5d-4885-bae2-98948b2ddedd\" (UID: \"770cd47b-7a5d-4885-bae2-98948b2ddedd\") " Nov 02 17:54:26.256878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.256197 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-openshift-ca" pod "770cd47b-7a5d-4885-bae2-98948b2ddedd" (UID: "770cd47b-7a5d-4885-bae2-98948b2ddedd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:26.257098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.257009 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:26.258360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.258333 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-xks-ca" pod "770cd47b-7a5d-4885-bae2-98948b2ddedd" (UID: "770cd47b-7a5d-4885-bae2-98948b2ddedd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:26.258849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.258827 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770cd47b-7a5d-4885-bae2-98948b2ddedd-kube-api-access-kfx8c" pod "770cd47b-7a5d-4885-bae2-98948b2ddedd" (UID: "770cd47b-7a5d-4885-bae2-98948b2ddedd"). InnerVolumeSpecName "kube-api-access-kfx8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:26.358716 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.358684 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfx8c\" (UniqueName: \"kubernetes.io/projected/770cd47b-7a5d-4885-bae2-98948b2ddedd-kube-api-access-kfx8c\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:26.358716 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.358719 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770cd47b-7a5d-4885-bae2-98948b2ddedd-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:26.442047 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.442014 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:26.752678 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:26.752646 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c519182_e8de_4e4a_8668_af65f7dea9b6.slice/crio-87fe93c17f93d2c386b9be39ad9f092fe25e6e5ac573a5b39777a55f614b7944 WatchSource:0}: Error finding container 87fe93c17f93d2c386b9be39ad9f092fe25e6e5ac573a5b39777a55f614b7944: Status 404 returned error can't find the container with id 87fe93c17f93d2c386b9be39ad9f092fe25e6e5ac573a5b39777a55f614b7944 Nov 02 17:54:26.755438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.755302 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7"] Nov 02 17:54:26.783667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:26.783633 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:27.083518 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.083425 4310 generic.go:356] "Generic (PLEG): container finished" podID="c3017323-41c0-4cde-9edb-ba9a6185351f" containerID="1cfbdab28178f0f79863a0218e1bd95bd980b935ca6aab8b75cfc5025382f2f0" exitCode=0 Nov 02 17:54:27.083680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.083509 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" event={"ID":"c3017323-41c0-4cde-9edb-ba9a6185351f","Type":"ContainerDied","Data":"1cfbdab28178f0f79863a0218e1bd95bd980b935ca6aab8b75cfc5025382f2f0"} Nov 02 17:54:27.085021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.085001 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" Nov 02 17:54:27.085257 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.085000 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r" event={"ID":"770cd47b-7a5d-4885-bae2-98948b2ddedd","Type":"ContainerDied","Data":"46669bf1a421913e79cb71e52e060f46629f94e64b6f7ee0d1eddb493baa38ba"} Nov 02 17:54:27.085257 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.085125 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46669bf1a421913e79cb71e52e060f46629f94e64b6f7ee0d1eddb493baa38ba" Nov 02 17:54:27.086793 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.086765 4310 generic.go:356] "Generic (PLEG): container finished" podID="35b62f9b-f0af-4474-a52e-5fb4a3c39e4d" containerID="b9da58ac8049143892414e5578bcb4c56ec6e8bafe036d94466118dddd2fbd58" exitCode=0 Nov 02 17:54:27.086929 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.086837 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" event={"ID":"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d","Type":"ContainerDied","Data":"b9da58ac8049143892414e5578bcb4c56ec6e8bafe036d94466118dddd2fbd58"} Nov 02 17:54:27.088582 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.088559 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" event={"ID":"4c519182-e8de-4e4a-8668-af65f7dea9b6","Type":"ContainerStarted","Data":"87fe93c17f93d2c386b9be39ad9f092fe25e6e5ac573a5b39777a55f614b7944"} Nov 02 17:54:27.247378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.247346 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:27.274330 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.274293 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-openshift-ca\") pod \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " Nov 02 17:54:27.274494 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.274344 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-kube-api-access-xnlm4\" (UniqueName: \"kubernetes.io/projected/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-kube-api-access-xnlm4\") pod \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " Nov 02 17:54:27.274561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.274495 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-xks-ca\") pod \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\" (UID: \"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d\") " Nov 02 17:54:27.277998 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.277963 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-xks-ca" pod "35b62f9b-f0af-4474-a52e-5fb4a3c39e4d" (UID: "35b62f9b-f0af-4474-a52e-5fb4a3c39e4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:27.278231 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.278207 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-openshift-ca" pod "35b62f9b-f0af-4474-a52e-5fb4a3c39e4d" (UID: "35b62f9b-f0af-4474-a52e-5fb4a3c39e4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:27.280026 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.280000 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-kube-api-access-xnlm4" pod "35b62f9b-f0af-4474-a52e-5fb4a3c39e4d" (UID: "35b62f9b-f0af-4474-a52e-5fb4a3c39e4d"). InnerVolumeSpecName "kube-api-access-xnlm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:27.376901 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.376873 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:27.376901 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.376901 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:27.377368 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.376917 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnlm4\" (UniqueName: \"kubernetes.io/projected/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d-kube-api-access-xnlm4\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:27.572331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:27.572297 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:28.098996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.098958 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" Nov 02 17:54:28.098996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.098973 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw" event={"ID":"35b62f9b-f0af-4474-a52e-5fb4a3c39e4d","Type":"ContainerDied","Data":"80b782e5dc218d7822a886f24f188c4427daaf382bf3ee2899c5fb1c483c0691"} Nov 02 17:54:28.099222 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.099028 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80b782e5dc218d7822a886f24f188c4427daaf382bf3ee2899c5fb1c483c0691" Nov 02 17:54:28.101202 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.101071 4310 generic.go:356] "Generic (PLEG): container finished" podID="4c519182-e8de-4e4a-8668-af65f7dea9b6" containerID="6b140c474efac7437eceb99948b09b12518b612aa3a3c8d0c606acdca592fc5d" exitCode=0 Nov 02 17:54:28.101202 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.101191 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" event={"ID":"4c519182-e8de-4e4a-8668-af65f7dea9b6","Type":"ContainerDied","Data":"6b140c474efac7437eceb99948b09b12518b612aa3a3c8d0c606acdca592fc5d"} Nov 02 17:54:28.272742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.272715 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:28.296567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.296537 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-openshift-ca\") pod \"c3017323-41c0-4cde-9edb-ba9a6185351f\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " Nov 02 17:54:28.296725 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.296700 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3017323-41c0-4cde-9edb-ba9a6185351f-kube-api-access-ndbz8\" (UniqueName: \"kubernetes.io/projected/c3017323-41c0-4cde-9edb-ba9a6185351f-kube-api-access-ndbz8\") pod \"c3017323-41c0-4cde-9edb-ba9a6185351f\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " Nov 02 17:54:28.296764 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.296737 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-xks-ca\") pod \"c3017323-41c0-4cde-9edb-ba9a6185351f\" (UID: \"c3017323-41c0-4cde-9edb-ba9a6185351f\") " Nov 02 17:54:28.297845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.297811 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-openshift-ca" pod "c3017323-41c0-4cde-9edb-ba9a6185351f" (UID: "c3017323-41c0-4cde-9edb-ba9a6185351f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:28.300283 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.300253 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3017323-41c0-4cde-9edb-ba9a6185351f-kube-api-access-ndbz8" pod "c3017323-41c0-4cde-9edb-ba9a6185351f" (UID: "c3017323-41c0-4cde-9edb-ba9a6185351f"). InnerVolumeSpecName "kube-api-access-ndbz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:28.300465 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.300442 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-xks-ca" pod "c3017323-41c0-4cde-9edb-ba9a6185351f" (UID: "c3017323-41c0-4cde-9edb-ba9a6185351f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:28.380090 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.380034 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:28.398916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.398880 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndbz8\" (UniqueName: \"kubernetes.io/projected/c3017323-41c0-4cde-9edb-ba9a6185351f-kube-api-access-ndbz8\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:28.398916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.398915 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:28.399141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.398931 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3017323-41c0-4cde-9edb-ba9a6185351f-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:28.803401 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.803367 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn"] Nov 02 17:54:28.804109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.804072 4310 cpu_manager.go:414] "containerMap: removing container" podUID="c3017323-41c0-4cde-9edb-ba9a6185351f" containerName="revoke-keys" Nov 02 17:54:28.804162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.804109 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3017323-41c0-4cde-9edb-ba9a6185351f" containerName="revoke-keys" Nov 02 17:54:28.804162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.804159 4310 cpu_manager.go:414] "containerMap: removing container" podUID="35b62f9b-f0af-4474-a52e-5fb4a3c39e4d" containerName="revoke-keys" Nov 02 17:54:28.804229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.804165 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b62f9b-f0af-4474-a52e-5fb4a3c39e4d" containerName="revoke-keys" Nov 02 17:54:28.804229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.804183 4310 cpu_manager.go:414] "containerMap: removing container" podUID="770cd47b-7a5d-4885-bae2-98948b2ddedd" containerName="revoke-keys" Nov 02 17:54:28.804229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.804188 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770cd47b-7a5d-4885-bae2-98948b2ddedd" containerName="revoke-keys" Nov 02 17:54:28.804326 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.804303 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3017323-41c0-4cde-9edb-ba9a6185351f" containerName="revoke-keys" Nov 02 17:54:28.804326 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.804318 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b62f9b-f0af-4474-a52e-5fb4a3c39e4d" containerName="revoke-keys" Nov 02 17:54:28.804385 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.804327 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="770cd47b-7a5d-4885-bae2-98948b2ddedd" containerName="revoke-keys" Nov 02 17:54:28.808595 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.808451 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:28.818370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.818287 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn"] Nov 02 17:54:28.921232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.921181 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e035996-zkkdn\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:28.921394 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.921314 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e035996-zkkdn\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:28.921487 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:28.921474 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-889tj\" (UniqueName: \"kubernetes.io/projected/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-kube-api-access-889tj\") pod \"maas-api-revoke-sub-0e035996-zkkdn\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:29.022960 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.022924 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e035996-zkkdn\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:29.023194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.023021 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e035996-zkkdn\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:29.023239 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.023194 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-889tj\" (UniqueName: \"kubernetes.io/projected/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-kube-api-access-889tj\") pod \"maas-api-revoke-sub-0e035996-zkkdn\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:29.023795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.023768 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e035996-zkkdn\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:29.025603 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.025575 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e035996-zkkdn\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:29.031367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.031337 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-889tj\" (UniqueName: \"kubernetes.io/projected/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-kube-api-access-889tj\") pod \"maas-api-revoke-sub-0e035996-zkkdn\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:29.111965 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.111925 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" event={"ID":"c3017323-41c0-4cde-9edb-ba9a6185351f","Type":"ContainerDied","Data":"98c39a32a789dd6809dcd77e8d1c5229be9d7f2fd3af4fc81fe63a842385241a"} Nov 02 17:54:29.111965 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.111954 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z" Nov 02 17:54:29.112194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.111986 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98c39a32a789dd6809dcd77e8d1c5229be9d7f2fd3af4fc81fe63a842385241a" Nov 02 17:54:29.130616 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.130587 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:29.285123 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.285031 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:29.322851 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.322817 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:29.329278 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.329244 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-openshift-ca\") pod \"4c519182-e8de-4e4a-8668-af65f7dea9b6\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " Nov 02 17:54:29.329624 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.329602 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c519182-e8de-4e4a-8668-af65f7dea9b6-kube-api-access-rlfvk\" (UniqueName: \"kubernetes.io/projected/4c519182-e8de-4e4a-8668-af65f7dea9b6-kube-api-access-rlfvk\") pod \"4c519182-e8de-4e4a-8668-af65f7dea9b6\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " Nov 02 17:54:29.329726 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.329709 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-xks-ca\") pod \"4c519182-e8de-4e4a-8668-af65f7dea9b6\" (UID: \"4c519182-e8de-4e4a-8668-af65f7dea9b6\") " Nov 02 17:54:29.331502 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.331470 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-openshift-ca" pod "4c519182-e8de-4e4a-8668-af65f7dea9b6" (UID: "4c519182-e8de-4e4a-8668-af65f7dea9b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:29.332866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.332834 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-xks-ca" pod "4c519182-e8de-4e4a-8668-af65f7dea9b6" (UID: "4c519182-e8de-4e4a-8668-af65f7dea9b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:29.334875 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.334848 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c519182-e8de-4e4a-8668-af65f7dea9b6-kube-api-access-rlfvk" pod "4c519182-e8de-4e4a-8668-af65f7dea9b6" (UID: "4c519182-e8de-4e4a-8668-af65f7dea9b6"). InnerVolumeSpecName "kube-api-access-rlfvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:29.480944 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.480907 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:29.481534 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.481514 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c519182-e8de-4e4a-8668-af65f7dea9b6-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:29.481639 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.481627 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlfvk\" (UniqueName: \"kubernetes.io/projected/4c519182-e8de-4e4a-8668-af65f7dea9b6-kube-api-access-rlfvk\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:29.521038 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:29.521007 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3145a2_86eb_49fb_8cd7_2a5f9ef7522c.slice/crio-07e7b8bf8242eae4408590accaa08887b1f2e3de6fce3ea576950d1648c25cd4 WatchSource:0}: Error finding container 07e7b8bf8242eae4408590accaa08887b1f2e3de6fce3ea576950d1648c25cd4: Status 404 returned error can't find the container with id 07e7b8bf8242eae4408590accaa08887b1f2e3de6fce3ea576950d1648c25cd4 Nov 02 17:54:29.523255 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:29.523231 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn"] Nov 02 17:54:30.122961 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:30.122869 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" event={"ID":"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c","Type":"ContainerStarted","Data":"51aba474f2321f5ed5051a22f6a51174d009baac7480b91394c4d4d745e98011"} Nov 02 17:54:30.122961 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:30.122923 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" event={"ID":"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c","Type":"ContainerStarted","Data":"07e7b8bf8242eae4408590accaa08887b1f2e3de6fce3ea576950d1648c25cd4"} Nov 02 17:54:30.124364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:30.124337 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" event={"ID":"4c519182-e8de-4e4a-8668-af65f7dea9b6","Type":"ContainerDied","Data":"87fe93c17f93d2c386b9be39ad9f092fe25e6e5ac573a5b39777a55f614b7944"} Nov 02 17:54:30.124364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:30.124356 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7" Nov 02 17:54:30.124575 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:30.124376 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87fe93c17f93d2c386b9be39ad9f092fe25e6e5ac573a5b39777a55f614b7944" Nov 02 17:54:30.147307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:30.147253 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" podStartSLOduration=2.147237404 podStartE2EDuration="2.147237404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:30.140629011 +0000 UTC m=+867.583711637" watchObservedRunningTime="2026-11-02 17:54:30.147237404 +0000 UTC m=+867.590320041" Nov 02 17:54:30.204967 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:30.204931 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:30.376152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:30.370616 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:30.465801 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:30.465773 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:30.995483 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:30.995445 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:31.165663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.164684 4310 generic.go:356] "Generic (PLEG): container finished" podID="fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c" containerID="51aba474f2321f5ed5051a22f6a51174d009baac7480b91394c4d4d745e98011" exitCode=0 Nov 02 17:54:31.165663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.164944 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" event={"ID":"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c","Type":"ContainerDied","Data":"51aba474f2321f5ed5051a22f6a51174d009baac7480b91394c4d4d745e98011"} Nov 02 17:54:31.201557 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.201513 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv"] Nov 02 17:54:31.202733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.202709 4310 cpu_manager.go:414] "containerMap: removing container" podUID="4c519182-e8de-4e4a-8668-af65f7dea9b6" containerName="revoke-keys" Nov 02 17:54:31.202814 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.202735 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c519182-e8de-4e4a-8668-af65f7dea9b6" containerName="revoke-keys" Nov 02 17:54:31.202925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.202913 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c519182-e8de-4e4a-8668-af65f7dea9b6" containerName="revoke-keys" Nov 02 17:54:31.210501 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.210472 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:31.220502 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.220465 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv"] Nov 02 17:54:31.363719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.363630 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ee2a68-wwtkv\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:31.363894 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.363859 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ee2a68-wwtkv\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:31.364014 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.363998 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56b82\" (UniqueName: \"kubernetes.io/projected/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-kube-api-access-56b82\") pod \"maas-api-revoke-sub-30ee2a68-wwtkv\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:31.466025 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.465985 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ee2a68-wwtkv\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:31.466202 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.466141 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56b82\" (UniqueName: \"kubernetes.io/projected/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-kube-api-access-56b82\") pod \"maas-api-revoke-sub-30ee2a68-wwtkv\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:31.466333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.466319 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ee2a68-wwtkv\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:31.466915 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.466897 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ee2a68-wwtkv\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:31.468452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.468434 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ee2a68-wwtkv\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:31.475612 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.475590 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56b82\" (UniqueName: \"kubernetes.io/projected/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-kube-api-access-56b82\") pod \"maas-api-revoke-sub-30ee2a68-wwtkv\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:31.547318 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.547257 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:31.781426 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.781373 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:31.909648 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:31.909584 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv"] Nov 02 17:54:32.175220 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.175185 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" event={"ID":"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69","Type":"ContainerStarted","Data":"ddfcabae1f90566fb2df7684bb48f6cb22055ebc7fbbb748126108667fede01d"} Nov 02 17:54:32.349559 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.349530 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:32.493537 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.493505 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-kube-api-access-889tj\" (UniqueName: \"kubernetes.io/projected/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-kube-api-access-889tj\") pod \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " Nov 02 17:54:32.493667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.493619 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-openshift-ca\") pod \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " Nov 02 17:54:32.493833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.493819 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-xks-ca\") pod \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\" (UID: \"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c\") " Nov 02 17:54:32.493966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.493940 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-openshift-ca" pod "fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c" (UID: "fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:32.494625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.494607 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:32.496403 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.496376 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-kube-api-access-889tj" pod "fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c" (UID: "fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c"). InnerVolumeSpecName "kube-api-access-889tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:32.496590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.496570 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-xks-ca" pod "fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c" (UID: "fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:32.559670 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.559628 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:32.596827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.596791 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:32.596827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:32.596822 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-889tj\" (UniqueName: \"kubernetes.io/projected/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c-kube-api-access-889tj\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:33.198667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:33.198569 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:33.199200 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:33.199174 4310 generic.go:356] "Generic (PLEG): container finished" podID="6e4a96e6-8cd1-4531-bf13-49de4b1e6c69" containerID="cc5062a8107408298db24d535bcc50648d6ef4467d22dbcce38012dfafbd180d" exitCode=0 Nov 02 17:54:33.199300 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:33.199256 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" event={"ID":"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69","Type":"ContainerDied","Data":"cc5062a8107408298db24d535bcc50648d6ef4467d22dbcce38012dfafbd180d"} Nov 02 17:54:33.200945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:33.200922 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" event={"ID":"fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c","Type":"ContainerDied","Data":"07e7b8bf8242eae4408590accaa08887b1f2e3de6fce3ea576950d1648c25cd4"} Nov 02 17:54:33.200945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:33.200937 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn" Nov 02 17:54:33.201133 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:33.200958 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07e7b8bf8242eae4408590accaa08887b1f2e3de6fce3ea576950d1648c25cd4" Nov 02 17:54:33.454473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:33.454389 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:33.524701 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:54:33.524661 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:54:33.980523 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:33.980486 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:34.390090 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:34.390056 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:34.462655 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:34.462620 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-openshift-ca\") pod \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " Nov 02 17:54:34.462852 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:34.462727 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-kube-api-access-56b82\" (UniqueName: \"kubernetes.io/projected/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-kube-api-access-56b82\") pod \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " Nov 02 17:54:34.463052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:34.463027 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-xks-ca\") pod \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\" (UID: \"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69\") " Nov 02 17:54:34.463401 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:34.463360 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-openshift-ca" pod "6e4a96e6-8cd1-4531-bf13-49de4b1e6c69" (UID: "6e4a96e6-8cd1-4531-bf13-49de4b1e6c69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:34.464139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:34.463895 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:34.466764 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:34.466727 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-kube-api-access-56b82" pod "6e4a96e6-8cd1-4531-bf13-49de4b1e6c69" (UID: "6e4a96e6-8cd1-4531-bf13-49de4b1e6c69"). InnerVolumeSpecName "kube-api-access-56b82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:34.467419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:34.467391 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-xks-ca" pod "6e4a96e6-8cd1-4531-bf13-49de4b1e6c69" (UID: "6e4a96e6-8cd1-4531-bf13-49de4b1e6c69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:34.565349 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:34.565257 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56b82\" (UniqueName: \"kubernetes.io/projected/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-kube-api-access-56b82\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:34.565349 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:34.565288 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:34.681334 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:34.681297 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:35.219212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:35.219181 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" Nov 02 17:54:35.219410 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:35.219219 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv" event={"ID":"6e4a96e6-8cd1-4531-bf13-49de4b1e6c69","Type":"ContainerDied","Data":"ddfcabae1f90566fb2df7684bb48f6cb22055ebc7fbbb748126108667fede01d"} Nov 02 17:54:35.219410 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:35.219274 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddfcabae1f90566fb2df7684bb48f6cb22055ebc7fbbb748126108667fede01d" Nov 02 17:54:35.383652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:35.383619 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:36.250427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:36.250393 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:37.058885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.058850 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4"] Nov 02 17:54:37.059671 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.059649 4310 cpu_manager.go:414] "containerMap: removing container" podUID="6e4a96e6-8cd1-4531-bf13-49de4b1e6c69" containerName="revoke-keys" Nov 02 17:54:37.059671 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.059671 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e4a96e6-8cd1-4531-bf13-49de4b1e6c69" containerName="revoke-keys" Nov 02 17:54:37.059834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.059686 4310 cpu_manager.go:414] "containerMap: removing container" podUID="fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c" containerName="revoke-keys" Nov 02 17:54:37.059834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.059692 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c" containerName="revoke-keys" Nov 02 17:54:37.059834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.059797 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e4a96e6-8cd1-4531-bf13-49de4b1e6c69" containerName="revoke-keys" Nov 02 17:54:37.059834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.059806 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c" containerName="revoke-keys" Nov 02 17:54:37.064018 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.063997 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:37.069634 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.069588 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:54:37.069822 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.069804 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:54:37.075909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.075881 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4"] Nov 02 17:54:37.150654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.150623 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2032de45-cfpb4\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:37.150827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.150745 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nrmb\" (UniqueName: \"kubernetes.io/projected/9b8eedda-d223-48e0-8c4e-9f109a62fd12-kube-api-access-9nrmb\") pod \"maas-api-revoke-sub-2032de45-cfpb4\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:37.150889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.150879 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2032de45-cfpb4\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:37.245823 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.245789 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:37.252048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.252020 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2032de45-cfpb4\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:37.252432 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.252129 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrmb\" (UniqueName: \"kubernetes.io/projected/9b8eedda-d223-48e0-8c4e-9f109a62fd12-kube-api-access-9nrmb\") pod \"maas-api-revoke-sub-2032de45-cfpb4\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:37.252432 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.252276 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2032de45-cfpb4\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:37.252888 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.252868 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2032de45-cfpb4\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:37.254427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.254407 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2032de45-cfpb4\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:37.262237 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.262217 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrmb\" (UniqueName: \"kubernetes.io/projected/9b8eedda-d223-48e0-8c4e-9f109a62fd12-kube-api-access-9nrmb\") pod \"maas-api-revoke-sub-2032de45-cfpb4\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:37.401228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.401133 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:37.807481 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:37.807453 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8eedda_d223_48e0_8c4e_9f109a62fd12.slice/crio-25a846a84acc69b2610fc69cd32b59c73dfcbd575a1d60dc19754a6fcfdd4d96 WatchSource:0}: Error finding container 25a846a84acc69b2610fc69cd32b59c73dfcbd575a1d60dc19754a6fcfdd4d96: Status 404 returned error can't find the container with id 25a846a84acc69b2610fc69cd32b59c73dfcbd575a1d60dc19754a6fcfdd4d96 Nov 02 17:54:37.810730 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.810703 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4"] Nov 02 17:54:37.981516 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.981477 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q"] Nov 02 17:54:37.986500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.986480 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:37.997360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:37.997334 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q"] Nov 02 17:54:38.072698 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.072670 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:38.115104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.104281 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6svzn\" (UniqueName: \"kubernetes.io/projected/12435051-cef8-4a9b-b749-b1c6ba61e6c9-kube-api-access-6svzn\") pod \"maas-api-revoke-sub-2042f8ad-br72q\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:38.115104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.104364 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2042f8ad-br72q\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:38.115104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.104527 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2042f8ad-br72q\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:38.277164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.271639 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2042f8ad-br72q\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:38.278371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.278340 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6svzn\" (UniqueName: \"kubernetes.io/projected/12435051-cef8-4a9b-b749-b1c6ba61e6c9-kube-api-access-6svzn\") pod \"maas-api-revoke-sub-2042f8ad-br72q\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:38.278611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.278591 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2042f8ad-br72q\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:38.279316 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.272762 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2042f8ad-br72q\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:38.283130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.282645 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2042f8ad-br72q\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:38.299128 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.298684 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6svzn\" (UniqueName: \"kubernetes.io/projected/12435051-cef8-4a9b-b749-b1c6ba61e6c9-kube-api-access-6svzn\") pod \"maas-api-revoke-sub-2042f8ad-br72q\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:38.302737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.301980 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc"] Nov 02 17:54:38.309610 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.308907 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:38.310360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.310176 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:38.314146 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.312364 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc"] Nov 02 17:54:38.317938 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.317870 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" event={"ID":"9b8eedda-d223-48e0-8c4e-9f109a62fd12","Type":"ContainerStarted","Data":"25a846a84acc69b2610fc69cd32b59c73dfcbd575a1d60dc19754a6fcfdd4d96"} Nov 02 17:54:38.382105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.381286 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8c41026-8bjgc\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:38.382105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.381500 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzcq5\" (UniqueName: \"kubernetes.io/projected/57ff0895-1873-4e12-b960-b38d6a746da8-kube-api-access-pzcq5\") pod \"maas-api-revoke-sub-b8c41026-8bjgc\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:38.382105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.381540 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8c41026-8bjgc\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:38.483452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.483418 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzcq5\" (UniqueName: \"kubernetes.io/projected/57ff0895-1873-4e12-b960-b38d6a746da8-kube-api-access-pzcq5\") pod \"maas-api-revoke-sub-b8c41026-8bjgc\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:38.483452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.483457 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8c41026-8bjgc\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:38.483662 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.483630 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8c41026-8bjgc\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:38.484262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.484237 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8c41026-8bjgc\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:38.486112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.486092 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8c41026-8bjgc\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:38.491728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.491707 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzcq5\" (UniqueName: \"kubernetes.io/projected/57ff0895-1873-4e12-b960-b38d6a746da8-kube-api-access-pzcq5\") pod \"maas-api-revoke-sub-b8c41026-8bjgc\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:38.609184 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:38.609153 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12435051_cef8_4a9b_b749_b1c6ba61e6c9.slice/crio-40792046f82631e18c24ed4623cc757ca3df7f00cc2aac051f2c69511afbb958 WatchSource:0}: Error finding container 40792046f82631e18c24ed4623cc757ca3df7f00cc2aac051f2c69511afbb958: Status 404 returned error can't find the container with id 40792046f82631e18c24ed4623cc757ca3df7f00cc2aac051f2c69511afbb958 Nov 02 17:54:38.614716 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.614688 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q"] Nov 02 17:54:38.675866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.675843 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:38.791028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.790989 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:38.995826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:38.995799 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc"] Nov 02 17:54:39.024629 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:39.022816 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ff0895_1873_4e12_b960_b38d6a746da8.slice/crio-ce819033a738d7cbfed774b2285f51b60df8a66014a525e7c3a1b6495d149aa1 WatchSource:0}: Error finding container ce819033a738d7cbfed774b2285f51b60df8a66014a525e7c3a1b6495d149aa1: Status 404 returned error can't find the container with id ce819033a738d7cbfed774b2285f51b60df8a66014a525e7c3a1b6495d149aa1 Nov 02 17:54:39.328656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:39.328618 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" event={"ID":"57ff0895-1873-4e12-b960-b38d6a746da8","Type":"ContainerStarted","Data":"ce819033a738d7cbfed774b2285f51b60df8a66014a525e7c3a1b6495d149aa1"} Nov 02 17:54:39.330547 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:39.330518 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" event={"ID":"12435051-cef8-4a9b-b749-b1c6ba61e6c9","Type":"ContainerStarted","Data":"a210a0914e9ba37eacb4e0ba7303fca8ff5654f32243190682d286ebacacabd3"} Nov 02 17:54:39.330703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:39.330563 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" event={"ID":"12435051-cef8-4a9b-b749-b1c6ba61e6c9","Type":"ContainerStarted","Data":"40792046f82631e18c24ed4623cc757ca3df7f00cc2aac051f2c69511afbb958"} Nov 02 17:54:39.332403 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:39.332376 4310 generic.go:356] "Generic (PLEG): container finished" podID="9b8eedda-d223-48e0-8c4e-9f109a62fd12" containerID="01d0fad1b830331de8a575812535a8ce477ee5859508638a2d6fa9d3958bf4a4" exitCode=0 Nov 02 17:54:39.332517 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:39.332412 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" event={"ID":"9b8eedda-d223-48e0-8c4e-9f109a62fd12","Type":"ContainerDied","Data":"01d0fad1b830331de8a575812535a8ce477ee5859508638a2d6fa9d3958bf4a4"} Nov 02 17:54:39.348403 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:39.348353 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" podStartSLOduration=2.348339861 podStartE2EDuration="2.348339861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:39.346726205 +0000 UTC m=+876.789808841" watchObservedRunningTime="2026-11-02 17:54:39.348339861 +0000 UTC m=+876.791422497" Nov 02 17:54:40.419342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:40.415301 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:40.540102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:40.535387 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" event={"ID":"57ff0895-1873-4e12-b960-b38d6a746da8","Type":"ContainerStarted","Data":"6bff4cedbf7ed08366da770e026da7b91dba9c390f9c942128346031c28d3f5e"} Nov 02 17:54:40.569105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:40.562925 4310 generic.go:356] "Generic (PLEG): container finished" podID="12435051-cef8-4a9b-b749-b1c6ba61e6c9" containerID="a210a0914e9ba37eacb4e0ba7303fca8ff5654f32243190682d286ebacacabd3" exitCode=0 Nov 02 17:54:40.569105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:40.563212 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" event={"ID":"12435051-cef8-4a9b-b749-b1c6ba61e6c9","Type":"ContainerDied","Data":"a210a0914e9ba37eacb4e0ba7303fca8ff5654f32243190682d286ebacacabd3"} Nov 02 17:54:40.646314 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:40.646206 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:40.673104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:40.647595 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:40.963251 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:40.963208 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:54:41.016247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.015177 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.120340 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.120184 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.303156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.293193 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:54:41.303156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.295325 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:54:41.365105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.352240 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.390110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.376215 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.616159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.599167 4310 generic.go:356] "Generic (PLEG): container finished" podID="57ff0895-1873-4e12-b960-b38d6a746da8" containerID="6bff4cedbf7ed08366da770e026da7b91dba9c390f9c942128346031c28d3f5e" exitCode=0 Nov 02 17:54:41.616159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.599636 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" event={"ID":"57ff0895-1873-4e12-b960-b38d6a746da8","Type":"ContainerDied","Data":"6bff4cedbf7ed08366da770e026da7b91dba9c390f9c942128346031c28d3f5e"} Nov 02 17:54:41.652104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.649174 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:41.652104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.649589 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:41.652104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.650245 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:54:41.652104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.650310 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:54:42.012159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.966180 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:42.012159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:41.966808 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:54:42.071523 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.071478 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:54:42.081247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.072407 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:54:42.181111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.158732 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:54:42.181111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.159356 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Nov 02 17:54:42.181111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.159794 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:54:42.550242 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.550190 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:54:42.754202 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.559564 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:54:42.811238 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.663236 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:54:43.105036 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.663608 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:54:43.208099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.749212 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:54:43.208099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.749285 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:43.208099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:42.967673 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.208099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:43.077350 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.208099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:43.201346 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:54:43.701180 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:43.653562 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded" Nov 02 17:54:43.785106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:43.761625 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.855342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:43.788207 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4v5sg" podUID="3b41c5ba-f619-4894-8e4f-427014637518" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:43.855342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:43.831600 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.259112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.240656 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.259112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.242131 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:54:44.259112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.242345 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:54:44.259112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.242494 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:44.259112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.242635 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:54:44.259112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.242790 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:54:44.285853 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.285705 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.362834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.334709 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" podUID="2144453b-de91-4404-8b54-5960b5a697ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.677720 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.677681 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp" podUID="d4955081-1a71-4933-8332-dd8bc035ae18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.678057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.678024 4310 patch_prober.go:29] interesting pod/thanos-querier-6c47fdcc77-94c69 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.13:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:44.678242 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.678220 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-6c47fdcc77-94c69" podUID="54365f2d-6ca2-4f59-8c06-c5406f9857e3" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.13:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:44.865105 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:54:44.829791 4310 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.109s" Nov 02 17:54:45.125606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.918370 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.151307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:44.998466 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:45.152575 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.039155 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jdv66" podUID="2144453b-de91-4404-8b54-5960b5a697ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.260462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.256224 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.260462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.257782 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:54:45.260462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.258111 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:54:45.260462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.258386 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:54:45.260462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.258646 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:45.260462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.258904 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:54:45.308607 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.307420 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.370017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.369985 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:45.396837 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.396787 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-openshift-ca\") pod \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " Nov 02 17:54:45.397241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.397213 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8eedda-d223-48e0-8c4e-9f109a62fd12-kube-api-access-9nrmb\" (UniqueName: \"kubernetes.io/projected/9b8eedda-d223-48e0-8c4e-9f109a62fd12-kube-api-access-9nrmb\") pod \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " Nov 02 17:54:45.398786 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.398762 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-xks-ca\") pod \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\" (UID: \"9b8eedda-d223-48e0-8c4e-9f109a62fd12\") " Nov 02 17:54:45.430935 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.430883 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-openshift-ca" pod "9b8eedda-d223-48e0-8c4e-9f109a62fd12" (UID: "9b8eedda-d223-48e0-8c4e-9f109a62fd12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:45.556915 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.453730 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:45.557617 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.454199 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Nov 02 17:54:45.612471 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.577878 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:45.738353 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.738306 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-openshift-ca\") pod \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " Nov 02 17:54:45.738785 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.738766 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12435051-cef8-4a9b-b749-b1c6ba61e6c9-kube-api-access-6svzn\" (UniqueName: \"kubernetes.io/projected/12435051-cef8-4a9b-b749-b1c6ba61e6c9-kube-api-access-6svzn\") pod \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " Nov 02 17:54:45.761421 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.738923 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-xks-ca\") pod \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\" (UID: \"12435051-cef8-4a9b-b749-b1c6ba61e6c9\") " Nov 02 17:54:45.787280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.787239 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-openshift-ca" pod "12435051-cef8-4a9b-b749-b1c6ba61e6c9" (UID: "12435051-cef8-4a9b-b749-b1c6ba61e6c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:45.906101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.890104 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:45.906101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.892607 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12435051-cef8-4a9b-b749-b1c6ba61e6c9-kube-api-access-6svzn" pod "12435051-cef8-4a9b-b749-b1c6ba61e6c9" (UID: "12435051-cef8-4a9b-b749-b1c6ba61e6c9"). InnerVolumeSpecName "kube-api-access-6svzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:45.926548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:45.911278 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-xks-ca" pod "12435051-cef8-4a9b-b749-b1c6ba61e6c9" (UID: "12435051-cef8-4a9b-b749-b1c6ba61e6c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:46.019106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.011710 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6svzn\" (UniqueName: \"kubernetes.io/projected/12435051-cef8-4a9b-b749-b1c6ba61e6c9-kube-api-access-6svzn\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:46.019106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.011755 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12435051-cef8-4a9b-b749-b1c6ba61e6c9-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:46.066264 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.066174 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.066498 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.066467 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" podUID="aed79116-9b99-4427-814f-e8af212f5126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.110228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.110191 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.114102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.110474 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.173129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.152175 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.173129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.152570 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.173129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.152615 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.193324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.186677 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" Nov 02 17:54:46.268134 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:54:46.262497 4310 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.148s" Nov 02 17:54:46.268134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.262776 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q" event={"ID":"12435051-cef8-4a9b-b749-b1c6ba61e6c9","Type":"ContainerDied","Data":"40792046f82631e18c24ed4623cc757ca3df7f00cc2aac051f2c69511afbb958"} Nov 02 17:54:46.268134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.262813 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40792046f82631e18c24ed4623cc757ca3df7f00cc2aac051f2c69511afbb958" Nov 02 17:54:46.268134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.262876 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:54:46.268134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.262910 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:54:46.268134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.262946 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:54:46.268134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.262998 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:54:46.439608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.404272 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.443550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.443508 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:54:46.471218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.459464 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8eedda-d223-48e0-8c4e-9f109a62fd12-kube-api-access-9nrmb" pod "9b8eedda-d223-48e0-8c4e-9f109a62fd12" (UID: "9b8eedda-d223-48e0-8c4e-9f109a62fd12"). InnerVolumeSpecName "kube-api-access-9nrmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:46.532226 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.484161 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:54:46.532226 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.484577 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:54:46.532226 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.484898 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:54:46.532226 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.485171 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:46.532226 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.485424 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:54:46.574118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.556253 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.574118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.556322 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.574118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.569170 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.574118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.569232 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.574652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.574618 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-xks-ca" pod "9b8eedda-d223-48e0-8c4e-9f109a62fd12" (UID: "9b8eedda-d223-48e0-8c4e-9f109a62fd12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:46.576544 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.576509 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.857104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.605710 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.880241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.880194 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.880499 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.617404 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:54:46.880929 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.615935 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nrmb\" (UniqueName: \"kubernetes.io/projected/9b8eedda-d223-48e0-8c4e-9f109a62fd12-kube-api-access-9nrmb\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:46.886243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.886220 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b8eedda-d223-48e0-8c4e-9f109a62fd12-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:46.886414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.821615 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Nov 02 17:54:46.886521 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.861950 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.886692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.886668 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.886783 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.861986 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.886877 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.886860 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.886961 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.862013 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:46.891113 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.887059 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:46.891278 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.654595 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" event={"ID":"9b8eedda-d223-48e0-8c4e-9f109a62fd12","Type":"ContainerDied","Data":"25a846a84acc69b2610fc69cd32b59c73dfcbd575a1d60dc19754a6fcfdd4d96"} Nov 02 17:54:46.891429 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.891413 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a846a84acc69b2610fc69cd32b59c73dfcbd575a1d60dc19754a6fcfdd4d96" Nov 02 17:54:46.943148 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:46.654828 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4" Nov 02 17:54:47.025301 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.025267 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:47.035109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.026011 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:47.066112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.020796 4310 patch_prober.go:29] interesting pod/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:47.066617 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.066383 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.131182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.104223 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:47.131182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.104301 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.131182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.105130 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-openshift-ca\") pod \"57ff0895-1873-4e12-b960-b38d6a746da8\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " Nov 02 17:54:47.131182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.105211 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ff0895-1873-4e12-b960-b38d6a746da8-kube-api-access-pzcq5\" (UniqueName: \"kubernetes.io/projected/57ff0895-1873-4e12-b960-b38d6a746da8-kube-api-access-pzcq5\") pod \"57ff0895-1873-4e12-b960-b38d6a746da8\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " Nov 02 17:54:47.131182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.105522 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-xks-ca\") pod \"57ff0895-1873-4e12-b960-b38d6a746da8\" (UID: \"57ff0895-1873-4e12-b960-b38d6a746da8\") " Nov 02 17:54:47.131182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.112536 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-openshift-ca" pod "57ff0895-1873-4e12-b960-b38d6a746da8" (UID: "57ff0895-1873-4e12-b960-b38d6a746da8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:47.164121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.164060 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:54:47.384550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.345900 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-xks-ca" pod "57ff0895-1873-4e12-b960-b38d6a746da8" (UID: "57ff0895-1873-4e12-b960-b38d6a746da8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:47.384550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.346050 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:47.384550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.346123 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.384550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.361351 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ff0895-1873-4e12-b960-b38d6a746da8-kube-api-access-pzcq5" pod "57ff0895-1873-4e12-b960-b38d6a746da8" (UID: "57ff0895-1873-4e12-b960-b38d6a746da8"). InnerVolumeSpecName "kube-api-access-pzcq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:47.384550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.362629 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.486592 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.427024 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:47.486592 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.427094 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57ff0895-1873-4e12-b960-b38d6a746da8-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:47.486592 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.427114 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzcq5\" (UniqueName: \"kubernetes.io/projected/57ff0895-1873-4e12-b960-b38d6a746da8-kube-api-access-pzcq5\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:47.486592 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.480669 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.486592 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.483123 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:47.486592 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.484241 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.653168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.639769 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:54:47.653168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.640319 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:54:47.653168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.640679 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:54:47.653168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.640998 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:47.653168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.641323 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:54:47.712987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.712946 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:47.900109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.894295 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.949154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.947329 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:47.949154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.947424 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.983114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.982366 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:47.983114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.982441 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.987331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.987300 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:47.987580 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.987553 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:47.987735 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.987323 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:47.987860 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:47.987841 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.020480 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.020442 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" event={"ID":"57ff0895-1873-4e12-b960-b38d6a746da8","Type":"ContainerDied","Data":"ce819033a738d7cbfed774b2285f51b60df8a66014a525e7c3a1b6495d149aa1"} Nov 02 17:54:48.020725 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.020705 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce819033a738d7cbfed774b2285f51b60df8a66014a525e7c3a1b6495d149aa1" Nov 02 17:54:48.025101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.021114 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc" Nov 02 17:54:48.124280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.124239 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:48.124570 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.124534 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.190129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.172447 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.382632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.382598 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:48.386227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.386184 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.522104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.511640 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:48.522104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.511718 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.832455 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.790904 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.871103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.818493 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.968243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.836250 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:54:48.968243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.836300 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.968243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.836323 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:54:48.968243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.836343 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:48.968243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.836361 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:54:48.968243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.867637 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:48.968243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.957588 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:48.968243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:48.939765 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:49.215173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.175798 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded" Nov 02 17:54:49.215173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.196144 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:49.215173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.213186 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:49.703213 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.701460 4310 patch_prober.go:29] interesting pod/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:49.703213 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.701557 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:49.856129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.843195 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:54:49.856129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.843261 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:49.856706 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.856653 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:54:49.901883 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:54:49.901845 4310 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8eedda_d223_48e0_8c4e_9f109a62fd12.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:54:49.977173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.965184 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:54:49.977173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.965482 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:54:49.977173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.965940 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:54:49.977173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.966292 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:49.977173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.966464 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:49.977173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.966613 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:49.977173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:49.966767 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:54:50.055374 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:50.055215 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:50.351156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:50.348019 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:54:50.867927 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:50.862929 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:50.970253 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:50.970204 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:50.975113 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:50.970837 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:54:51.044099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:50.975352 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:51.044099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:50.975387 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:54:51.044666 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:50.975407 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:54:51.062767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:50.975426 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:54:51.062767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:50.975452 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:54:51.237661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.235897 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7716-e49fca"] Nov 02 17:54:51.237661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.237021 4310 cpu_manager.go:414] "containerMap: removing container" podUID="12435051-cef8-4a9b-b749-b1c6ba61e6c9" containerName="revoke-keys" Nov 02 17:54:51.237661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.237045 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12435051-cef8-4a9b-b749-b1c6ba61e6c9" containerName="revoke-keys" Nov 02 17:54:51.237661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.237066 4310 cpu_manager.go:414] "containerMap: removing container" podUID="57ff0895-1873-4e12-b960-b38d6a746da8" containerName="revoke-keys" Nov 02 17:54:51.237661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.237093 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ff0895-1873-4e12-b960-b38d6a746da8" containerName="revoke-keys" Nov 02 17:54:51.237661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.237107 4310 cpu_manager.go:414] "containerMap: removing container" podUID="9b8eedda-d223-48e0-8c4e-9f109a62fd12" containerName="revoke-keys" Nov 02 17:54:51.237661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.237113 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8eedda-d223-48e0-8c4e-9f109a62fd12" containerName="revoke-keys" Nov 02 17:54:51.260104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.256001 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="12435051-cef8-4a9b-b749-b1c6ba61e6c9" containerName="revoke-keys" Nov 02 17:54:51.260104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.256034 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ff0895-1873-4e12-b960-b38d6a746da8" containerName="revoke-keys" Nov 02 17:54:51.260104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.256048 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8eedda-d223-48e0-8c4e-9f109a62fd12" containerName="revoke-keys" Nov 02 17:54:51.307371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.289675 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:51.330118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.327892 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7716-e49fca"] Nov 02 17:54:51.330118 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.328027 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-e49fca" Nov 02 17:54:51.348097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.336378 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pqt9j\"" Nov 02 17:54:51.367582 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.350297 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:51.457174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.452599 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfh5w\" (UniqueName: \"kubernetes.io/projected/845a14c4-35c9-4e23-878e-b0908db83c42-kube-api-access-qfh5w\") pod \"test-curl-7716-e49fca\" (UID: \"845a14c4-35c9-4e23-878e-b0908db83c42\") " pod="opendatahub/test-curl-7716-e49fca" Nov 02 17:54:51.461027 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.460986 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:51.611487 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.611398 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfh5w\" (UniqueName: \"kubernetes.io/projected/845a14c4-35c9-4e23-878e-b0908db83c42-kube-api-access-qfh5w\") pod \"test-curl-7716-e49fca\" (UID: \"845a14c4-35c9-4e23-878e-b0908db83c42\") " pod="opendatahub/test-curl-7716-e49fca" Nov 02 17:54:51.857144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.847037 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:54:51.866156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.863502 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:51.940719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.939251 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:51.998698 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:51.990432 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:54:52.100157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.072809 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:52.100157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.073230 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:52.100157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.074868 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:52.100157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.082271 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:54:52.137170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.101579 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:54:52.140106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.137459 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Nov 02 17:54:52.146229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.142708 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfh5w\" (UniqueName: \"kubernetes.io/projected/845a14c4-35c9-4e23-878e-b0908db83c42-kube-api-access-qfh5w\") pod \"test-curl-7716-e49fca\" (UID: \"845a14c4-35c9-4e23-878e-b0908db83c42\") " pod="opendatahub/test-curl-7716-e49fca" Nov 02 17:54:52.184237 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.184181 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:54:52.188204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.188017 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:52.207547 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.207527 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:54:52.209549 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.209530 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:54:52.293970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.293940 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:54:52.330149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.319148 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-e49fca" Nov 02 17:54:52.484175 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.479859 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v"] Nov 02 17:54:52.492094 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.486538 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:52.503435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.495015 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:54:52.533104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.533038 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v"] Nov 02 17:54:52.535360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.533315 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:54:52.535360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.533359 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:54:52.551264 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.551226 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:54:52.555825 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.555160 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:54:52.558212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.558184 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:54:52.582540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.582516 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:54:52.640461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.640438 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb309a27-bdm9v\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:52.643106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.642932 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb309a27-bdm9v\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:52.643395 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.643373 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpzvl\" (UniqueName: \"kubernetes.io/projected/481151a8-e4d4-4a0f-ae48-68102f129a23-kube-api-access-dpzvl\") pod \"maas-api-revoke-sub-eb309a27-bdm9v\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:52.746828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.746760 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb309a27-bdm9v\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:52.747111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.747095 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpzvl\" (UniqueName: \"kubernetes.io/projected/481151a8-e4d4-4a0f-ae48-68102f129a23-kube-api-access-dpzvl\") pod \"maas-api-revoke-sub-eb309a27-bdm9v\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:52.747334 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.747322 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb309a27-bdm9v\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:52.748465 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.748435 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb309a27-bdm9v\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:52.755633 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.755594 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb309a27-bdm9v\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:52.760541 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.760514 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpzvl\" (UniqueName: \"kubernetes.io/projected/481151a8-e4d4-4a0f-ae48-68102f129a23-kube-api-access-dpzvl\") pod \"maas-api-revoke-sub-eb309a27-bdm9v\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:52.870805 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.870713 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7716-e49fca"] Nov 02 17:54:52.876945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.876924 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:52.928997 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:52.928965 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:53.045953 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:53.045879 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:53.183788 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:53.183759 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481151a8_e4d4_4a0f_ae48_68102f129a23.slice/crio-019abfc0347de08d7660120c768fe0b566387bdb1dc77053ad31a3cee759fdff WatchSource:0}: Error finding container 019abfc0347de08d7660120c768fe0b566387bdb1dc77053ad31a3cee759fdff: Status 404 returned error can't find the container with id 019abfc0347de08d7660120c768fe0b566387bdb1dc77053ad31a3cee759fdff Nov 02 17:54:53.186622 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:53.186600 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v"] Nov 02 17:54:53.211379 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:53.211339 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:53.313959 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:53.313858 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:53.689089 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:53.689033 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:53.695590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:53.695456 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7716-e49fca" event={"ID":"845a14c4-35c9-4e23-878e-b0908db83c42","Type":"ContainerStarted","Data":"f1b397dab502f12cb5041b2b92b33a2a796b3361de6bda6be2921b1a164ab940"} Nov 02 17:54:53.698834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:53.698805 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" event={"ID":"481151a8-e4d4-4a0f-ae48-68102f129a23","Type":"ContainerStarted","Data":"e2a325bda07adb7feb4ebecf1f9d2849a6c2178d053249ac45792f659a3ba6fa"} Nov 02 17:54:53.699043 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:53.699024 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" event={"ID":"481151a8-e4d4-4a0f-ae48-68102f129a23","Type":"ContainerStarted","Data":"019abfc0347de08d7660120c768fe0b566387bdb1dc77053ad31a3cee759fdff"} Nov 02 17:54:53.837820 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:53.837784 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:53.888536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:53.876745 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" podStartSLOduration=1.876722099 podStartE2EDuration="1.876722099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:53.728699475 +0000 UTC m=+891.171782113" watchObservedRunningTime="2026-11-02 17:54:53.876722099 +0000 UTC m=+891.319804738" Nov 02 17:54:54.511406 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:54.511320 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:54.732826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:54.732773 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7716-e49fca" event={"ID":"845a14c4-35c9-4e23-878e-b0908db83c42","Type":"ContainerStarted","Data":"41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f"} Nov 02 17:54:54.734896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:54.734864 4310 generic.go:356] "Generic (PLEG): container finished" podID="481151a8-e4d4-4a0f-ae48-68102f129a23" containerID="e2a325bda07adb7feb4ebecf1f9d2849a6c2178d053249ac45792f659a3ba6fa" exitCode=0 Nov 02 17:54:54.735057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:54.734947 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" event={"ID":"481151a8-e4d4-4a0f-ae48-68102f129a23","Type":"ContainerDied","Data":"e2a325bda07adb7feb4ebecf1f9d2849a6c2178d053249ac45792f659a3ba6fa"} Nov 02 17:54:54.753971 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:54.753917 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7716-e49fca" podStartSLOduration=6.021135986 podStartE2EDuration="6.7538983s" totalImagesPullingTime="732.762314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:54.747187977 +0000 UTC m=+892.190270613" watchObservedRunningTime="2026-11-02 17:54:54.7538983 +0000 UTC m=+892.196980932" Nov 02 17:54:55.438706 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.438669 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7716-e49fca"] Nov 02 17:54:55.454638 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.454606 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7716-e49fca"] Nov 02 17:54:55.480100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.476493 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:55.580100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.579932 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:55.860145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.853565 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk"] Nov 02 17:54:55.860145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.855661 4310 cpu_manager.go:414] "containerMap: removing container" podUID="845a14c4-35c9-4e23-878e-b0908db83c42" containerName="test-curl-7716-e49fca" Nov 02 17:54:55.860145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.855684 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845a14c4-35c9-4e23-878e-b0908db83c42" containerName="test-curl-7716-e49fca" Nov 02 17:54:55.860145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.856763 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="845a14c4-35c9-4e23-878e-b0908db83c42" containerName="test-curl-7716-e49fca" Nov 02 17:54:55.870125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.866729 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:55.870125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.868201 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk"] Nov 02 17:54:55.873973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.873945 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d59fa283-openshift-default-dockercfg-2z5jl\"" Nov 02 17:54:55.885210 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.885181 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:55.940012 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.939977 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-podinfo\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:55.940170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.940045 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nc5r\" (UniqueName: \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-kube-api-access-5nc5r\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:55.940237 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.940150 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-socket\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:55.940302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.940284 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-data\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:55.940358 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.940334 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-credential-socket\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:55.940424 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.940406 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-envoy\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:55.940575 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.940557 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-certs\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:55.940608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.940590 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-token\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:55.940654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.940616 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istiod-ca-cert\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:55.952497 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.952476 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:55.995125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:55.995070 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:56.041418 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.041340 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481151a8-e4d4-4a0f-ae48-68102f129a23-kube-api-access-dpzvl\" (UniqueName: \"kubernetes.io/projected/481151a8-e4d4-4a0f-ae48-68102f129a23-kube-api-access-dpzvl\") pod \"481151a8-e4d4-4a0f-ae48-68102f129a23\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " Nov 02 17:54:56.041654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.041629 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-xks-ca\") pod \"481151a8-e4d4-4a0f-ae48-68102f129a23\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " Nov 02 17:54:56.041836 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.041818 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-openshift-ca\") pod \"481151a8-e4d4-4a0f-ae48-68102f129a23\" (UID: \"481151a8-e4d4-4a0f-ae48-68102f129a23\") " Nov 02 17:54:56.042176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.042131 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-certs\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.042176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.042173 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-token\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.042341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.042205 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istiod-ca-cert\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.042341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.042222 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-openshift-ca" pod "481151a8-e4d4-4a0f-ae48-68102f129a23" (UID: "481151a8-e4d4-4a0f-ae48-68102f129a23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:56.042489 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.042411 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-podinfo\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.042489 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.042483 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nc5r\" (UniqueName: \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-kube-api-access-5nc5r\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.042612 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.042561 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-socket\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.042682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.042662 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-data\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.042832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.042722 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-credential-socket\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.042832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.042793 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-envoy\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.043026 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.042987 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istiod-ca-cert\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.043523 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.043420 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-credential-socket\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.043523 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.043447 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-data\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.043523 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.043465 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-socket\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.043739 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.043708 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-certs\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.044128 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.043929 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481151a8-e4d4-4a0f-ae48-68102f129a23-kube-api-access-dpzvl" pod "481151a8-e4d4-4a0f-ae48-68102f129a23" (UID: "481151a8-e4d4-4a0f-ae48-68102f129a23"). InnerVolumeSpecName "kube-api-access-dpzvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:56.044128 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.044072 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-xks-ca" pod "481151a8-e4d4-4a0f-ae48-68102f129a23" (UID: "481151a8-e4d4-4a0f-ae48-68102f129a23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:54:56.044282 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.044129 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:56.045840 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.045815 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-podinfo\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.046144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.046123 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-envoy\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.052262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.052242 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-token\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.052494 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.052478 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nc5r\" (UniqueName: \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-kube-api-access-5nc5r\") pod \"e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.175100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.174595 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:56.186102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.175585 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481151a8-e4d4-4a0f-ae48-68102f129a23-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:56.186102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.175619 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpzvl\" (UniqueName: \"kubernetes.io/projected/481151a8-e4d4-4a0f-ae48-68102f129a23-kube-api-access-dpzvl\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:56.206693 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.205308 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:56.312777 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.312744 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:56.465050 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.464843 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:56.736989 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:54:56.736963 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfc8f71_82b9_4c5e_b7e4_b7d6cce90ddc.slice/crio-355716f335b3a0ee6055d864881935d65e2c8f7fb707d85f1d55924a65cc9831 WatchSource:0}: Error finding container 355716f335b3a0ee6055d864881935d65e2c8f7fb707d85f1d55924a65cc9831: Status 404 returned error can't find the container with id 355716f335b3a0ee6055d864881935d65e2c8f7fb707d85f1d55924a65cc9831 Nov 02 17:54:56.740125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.740097 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:54:56.740204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.740162 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:54:56.740204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.740190 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:54:56.740635 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.740578 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk"] Nov 02 17:54:56.757252 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.757228 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" Nov 02 17:54:56.757374 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.757269 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v" event={"ID":"481151a8-e4d4-4a0f-ae48-68102f129a23","Type":"ContainerDied","Data":"019abfc0347de08d7660120c768fe0b566387bdb1dc77053ad31a3cee759fdff"} Nov 02 17:54:56.757374 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.757329 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="019abfc0347de08d7660120c768fe0b566387bdb1dc77053ad31a3cee759fdff" Nov 02 17:54:56.767033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.766991 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" event={"ID":"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc","Type":"ContainerStarted","Data":"355716f335b3a0ee6055d864881935d65e2c8f7fb707d85f1d55924a65cc9831"} Nov 02 17:54:56.767189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.767125 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7716-e49fca" podUID="845a14c4-35c9-4e23-878e-b0908db83c42" containerName="test-curl-7716-e49fca" containerID="cri-o://41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f" gracePeriod=2 Nov 02 17:54:56.937316 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:56.937284 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:57.099950 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:57.099913 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:57.250106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:57.249647 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:57.778447 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:57.778413 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" event={"ID":"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc","Type":"ContainerStarted","Data":"2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c"} Nov 02 17:54:57.811806 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:57.811757 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" podStartSLOduration=2.81174359 podStartE2EDuration="2.81174359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:57.807725966 +0000 UTC m=+895.250808604" watchObservedRunningTime="2026-11-02 17:54:57.81174359 +0000 UTC m=+895.254826226" Nov 02 17:54:58.209554 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:58.209472 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:58.216326 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:58.216298 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:58.334632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:58.334599 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:58.788344 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:58.788316 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:58.789297 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:58.789274 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:54:59.032172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.032107 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-e49fca" Nov 02 17:54:59.180977 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.180943 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845a14c4-35c9-4e23-878e-b0908db83c42-kube-api-access-qfh5w\" (UniqueName: \"kubernetes.io/projected/845a14c4-35c9-4e23-878e-b0908db83c42-kube-api-access-qfh5w\") pod \"845a14c4-35c9-4e23-878e-b0908db83c42\" (UID: \"845a14c4-35c9-4e23-878e-b0908db83c42\") " Nov 02 17:54:59.183116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.183053 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845a14c4-35c9-4e23-878e-b0908db83c42-kube-api-access-qfh5w" pod "845a14c4-35c9-4e23-878e-b0908db83c42" (UID: "845a14c4-35c9-4e23-878e-b0908db83c42"). InnerVolumeSpecName "kube-api-access-qfh5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:59.282919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.282831 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfh5w\" (UniqueName: \"kubernetes.io/projected/845a14c4-35c9-4e23-878e-b0908db83c42-kube-api-access-qfh5w\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:54:59.459439 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.459401 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:59.606556 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.606472 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:54:59.813102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.810180 4310 generic.go:356] "Generic (PLEG): container finished" podID="845a14c4-35c9-4e23-878e-b0908db83c42" containerID="41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f" exitCode=137 Nov 02 17:54:59.813102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.811508 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-e49fca" Nov 02 17:54:59.821104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.816317 4310 scope.go:122] "RemoveContainer" containerID="41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f" Nov 02 17:54:59.870142 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:54:59.869415 4310 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7716-e49fca\" is forbidden: User \"system:node:ip-10-0-135-9.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-9.us-east-2.compute.internal' and this object" podUID="845a14c4-35c9-4e23-878e-b0908db83c42" pod="opendatahub/test-curl-7716-e49fca" Nov 02 17:54:59.881396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.879367 4310 scope.go:122] "RemoveContainer" containerID="41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f" Nov 02 17:54:59.881396 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:54:59.879785 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f\": container with ID starting with 41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f not found: ID does not exist" containerID="41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f" Nov 02 17:54:59.881396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.879831 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f"} err="failed to get container status \"41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f\": rpc error: code = NotFound desc = could not find container \"41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f\": container with ID starting with 41b465d1886a2fb6dfb42ccb60d47d439eaae88847633eb81d79d07f904cc42f not found: ID does not exist" Nov 02 17:54:59.996895 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:54:59.996844 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:00.146980 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.146865 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:00.165377 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.165342 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w"] Nov 02 17:55:00.166111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.166095 4310 cpu_manager.go:414] "containerMap: removing container" podUID="481151a8-e4d4-4a0f-ae48-68102f129a23" containerName="revoke-keys" Nov 02 17:55:00.166172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.166114 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481151a8-e4d4-4a0f-ae48-68102f129a23" containerName="revoke-keys" Nov 02 17:55:00.166290 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.166278 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="481151a8-e4d4-4a0f-ae48-68102f129a23" containerName="revoke-keys" Nov 02 17:55:00.171738 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.171716 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:00.178021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.177993 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:55:00.178187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.178040 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:55:00.185703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.185675 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w"] Nov 02 17:55:00.198690 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:55:00.198644 4310 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7716-e49fca\" is forbidden: User \"system:node:ip-10-0-135-9.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-9.us-east-2.compute.internal' and this object" podUID="845a14c4-35c9-4e23-878e-b0908db83c42" pod="opendatahub/test-curl-7716-e49fca" Nov 02 17:55:00.296680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.296619 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:00.308110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.308062 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93c75a54-wdn2w\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:00.308303 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.308222 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93c75a54-wdn2w\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:00.308303 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.308284 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nbj2\" (UniqueName: \"kubernetes.io/projected/cc7d6535-bc17-41ad-bd97-87013fa13171-kube-api-access-7nbj2\") pod \"maas-api-revoke-sub-93c75a54-wdn2w\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:00.410154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.410037 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93c75a54-wdn2w\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:00.410342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.410202 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93c75a54-wdn2w\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:00.410342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.410276 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nbj2\" (UniqueName: \"kubernetes.io/projected/cc7d6535-bc17-41ad-bd97-87013fa13171-kube-api-access-7nbj2\") pod \"maas-api-revoke-sub-93c75a54-wdn2w\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:00.411053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.411016 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93c75a54-wdn2w\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:00.412790 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.412752 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93c75a54-wdn2w\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:00.423691 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.423658 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nbj2\" (UniqueName: \"kubernetes.io/projected/cc7d6535-bc17-41ad-bd97-87013fa13171-kube-api-access-7nbj2\") pod \"maas-api-revoke-sub-93c75a54-wdn2w\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:00.489931 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.489891 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:00.707187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.706633 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845a14c4-35c9-4e23-878e-b0908db83c42" path="/var/lib/kubelet/pods/845a14c4-35c9-4e23-878e-b0908db83c42/volumes" Nov 02 17:55:00.784563 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.784524 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:00.808141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:00.803155 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w"] Nov 02 17:55:01.294647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:01.294613 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:01.423161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:01.418843 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:02.091545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:02.091512 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" event={"ID":"cc7d6535-bc17-41ad-bd97-87013fa13171","Type":"ContainerStarted","Data":"a25e167564caf64a83ad54b12330bd85872b10f0d24fa83e01ce918f9e83829d"} Nov 02 17:55:02.091695 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:02.091570 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" event={"ID":"cc7d6535-bc17-41ad-bd97-87013fa13171","Type":"ContainerStarted","Data":"463efee26ad02c1bcba92b37cbc89beca9eeb50242405814eb7cd000423b001b"} Nov 02 17:55:02.114825 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:02.114770 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" podStartSLOduration=2.114754311 podStartE2EDuration="2.114754311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:02.112928486 +0000 UTC m=+899.556011136" watchObservedRunningTime="2026-11-02 17:55:02.114754311 +0000 UTC m=+899.557836947" Nov 02 17:55:02.154345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:02.154315 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:03.103055 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:03.102958 4310 generic.go:356] "Generic (PLEG): container finished" podID="cc7d6535-bc17-41ad-bd97-87013fa13171" containerID="a25e167564caf64a83ad54b12330bd85872b10f0d24fa83e01ce918f9e83829d" exitCode=0 Nov 02 17:55:03.103526 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:03.103048 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" event={"ID":"cc7d6535-bc17-41ad-bd97-87013fa13171","Type":"ContainerDied","Data":"a25e167564caf64a83ad54b12330bd85872b10f0d24fa83e01ce918f9e83829d"} Nov 02 17:55:03.174997 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:03.174961 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:03.525067 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:55:03.525029 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:55:03.743093 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:03.743042 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:03.847225 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:03.847142 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:04.073748 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.073707 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:04.431740 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.431711 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:04.494975 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.494942 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc7d6535-bc17-41ad-bd97-87013fa13171-kube-api-access-7nbj2\" (UniqueName: \"kubernetes.io/projected/cc7d6535-bc17-41ad-bd97-87013fa13171-kube-api-access-7nbj2\") pod \"cc7d6535-bc17-41ad-bd97-87013fa13171\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " Nov 02 17:55:04.495256 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.495237 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-openshift-ca\") pod \"cc7d6535-bc17-41ad-bd97-87013fa13171\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " Nov 02 17:55:04.495316 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.495272 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-xks-ca\") pod \"cc7d6535-bc17-41ad-bd97-87013fa13171\" (UID: \"cc7d6535-bc17-41ad-bd97-87013fa13171\") " Nov 02 17:55:04.495573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.495552 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-openshift-ca" pod "cc7d6535-bc17-41ad-bd97-87013fa13171" (UID: "cc7d6535-bc17-41ad-bd97-87013fa13171"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:04.496144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.496122 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:04.497374 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.497338 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7d6535-bc17-41ad-bd97-87013fa13171-kube-api-access-7nbj2" pod "cc7d6535-bc17-41ad-bd97-87013fa13171" (UID: "cc7d6535-bc17-41ad-bd97-87013fa13171"). InnerVolumeSpecName "kube-api-access-7nbj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:04.497485 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.497440 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-xks-ca" pod "cc7d6535-bc17-41ad-bd97-87013fa13171" (UID: "cc7d6535-bc17-41ad-bd97-87013fa13171"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:04.597798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.597763 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nbj2\" (UniqueName: \"kubernetes.io/projected/cc7d6535-bc17-41ad-bd97-87013fa13171-kube-api-access-7nbj2\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:04.597798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.597800 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7d6535-bc17-41ad-bd97-87013fa13171-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:04.968579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:04.968539 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:05.097104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:05.093112 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:05.174410 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:05.174381 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" Nov 02 17:55:05.180483 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:05.180445 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w" event={"ID":"cc7d6535-bc17-41ad-bd97-87013fa13171","Type":"ContainerDied","Data":"463efee26ad02c1bcba92b37cbc89beca9eeb50242405814eb7cd000423b001b"} Nov 02 17:55:05.180828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:05.180810 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="463efee26ad02c1bcba92b37cbc89beca9eeb50242405814eb7cd000423b001b" Nov 02 17:55:05.302806 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:05.302727 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:05.972295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:05.972255 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:06.026756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.026718 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8"] Nov 02 17:55:06.027550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.027532 4310 cpu_manager.go:414] "containerMap: removing container" podUID="cc7d6535-bc17-41ad-bd97-87013fa13171" containerName="revoke-keys" Nov 02 17:55:06.027628 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.027551 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc7d6535-bc17-41ad-bd97-87013fa13171" containerName="revoke-keys" Nov 02 17:55:06.027683 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.027673 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc7d6535-bc17-41ad-bd97-87013fa13171" containerName="revoke-keys" Nov 02 17:55:06.037196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.037166 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:06.042970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.042943 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-be1b0f58\"" Nov 02 17:55:06.050185 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.050156 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8"] Nov 02 17:55:06.061386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.061363 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-be1b0f58-dockercfg-ghhh6\"" Nov 02 17:55:06.082566 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.082518 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:06.086713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.086693 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h"] Nov 02 17:55:06.091529 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.091511 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:06.120344 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.120230 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h"] Nov 02 17:55:06.292568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.292476 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1103c0e1-74d5-4035-ba22-f56bd12f16dd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h\" (UID: \"1103c0e1-74d5-4035-ba22-f56bd12f16dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:06.304406 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.304368 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fbpw\" (UniqueName: \"kubernetes.io/projected/a890090c-9e21-4253-a027-05438d80f288-kube-api-access-7fbpw\") pod \"payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8\" (UID: \"a890090c-9e21-4253-a027-05438d80f288\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:06.322110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.314400 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a890090c-9e21-4253-a027-05438d80f288-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8\" (UID: \"a890090c-9e21-4253-a027-05438d80f288\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:06.421343 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.421305 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jpst\" (UniqueName: \"kubernetes.io/projected/1103c0e1-74d5-4035-ba22-f56bd12f16dd-kube-api-access-2jpst\") pod \"payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h\" (UID: \"1103c0e1-74d5-4035-ba22-f56bd12f16dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:06.540278 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.539524 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1103c0e1-74d5-4035-ba22-f56bd12f16dd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h\" (UID: \"1103c0e1-74d5-4035-ba22-f56bd12f16dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:06.542331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.542295 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fbpw\" (UniqueName: \"kubernetes.io/projected/a890090c-9e21-4253-a027-05438d80f288-kube-api-access-7fbpw\") pod \"payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8\" (UID: \"a890090c-9e21-4253-a027-05438d80f288\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:06.542822 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.542758 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a890090c-9e21-4253-a027-05438d80f288-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8\" (UID: \"a890090c-9e21-4253-a027-05438d80f288\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:06.553991 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.553961 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jpst\" (UniqueName: \"kubernetes.io/projected/1103c0e1-74d5-4035-ba22-f56bd12f16dd-kube-api-access-2jpst\") pod \"payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h\" (UID: \"1103c0e1-74d5-4035-ba22-f56bd12f16dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:06.555484 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.555452 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1103c0e1-74d5-4035-ba22-f56bd12f16dd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h\" (UID: \"1103c0e1-74d5-4035-ba22-f56bd12f16dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:06.560664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.560634 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a890090c-9e21-4253-a027-05438d80f288-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8\" (UID: \"a890090c-9e21-4253-a027-05438d80f288\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:06.579664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.569310 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jpst\" (UniqueName: \"kubernetes.io/projected/1103c0e1-74d5-4035-ba22-f56bd12f16dd-kube-api-access-2jpst\") pod \"payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h\" (UID: \"1103c0e1-74d5-4035-ba22-f56bd12f16dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:06.588192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.579881 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fbpw\" (UniqueName: \"kubernetes.io/projected/a890090c-9e21-4253-a027-05438d80f288-kube-api-access-7fbpw\") pod \"payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8\" (UID: \"a890090c-9e21-4253-a027-05438d80f288\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:06.659629 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.659591 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:06.712091 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.712047 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:06.956923 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:06.956871 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda890090c_9e21_4253_a027_05438d80f288.slice/crio-44b9a3089769e5c78ae07d82c3bdb2f601daa5d00709ce54bfb9903bc66f9ea5 WatchSource:0}: Error finding container 44b9a3089769e5c78ae07d82c3bdb2f601daa5d00709ce54bfb9903bc66f9ea5: Status 404 returned error can't find the container with id 44b9a3089769e5c78ae07d82c3bdb2f601daa5d00709ce54bfb9903bc66f9ea5 Nov 02 17:55:06.959437 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.959405 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8"] Nov 02 17:55:06.973451 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:06.973424 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:07.148588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:07.146439 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h"] Nov 02 17:55:07.148588 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:07.148493 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1103c0e1_74d5_4035_ba22_f56bd12f16dd.slice/crio-f8fdffa575fd154056035b1befafa4e94eb0ea839c22c111fabeca9a0570d361 WatchSource:0}: Error finding container f8fdffa575fd154056035b1befafa4e94eb0ea839c22c111fabeca9a0570d361: Status 404 returned error can't find the container with id f8fdffa575fd154056035b1befafa4e94eb0ea839c22c111fabeca9a0570d361 Nov 02 17:55:07.452680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:07.452581 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" event={"ID":"a890090c-9e21-4253-a027-05438d80f288","Type":"ContainerStarted","Data":"c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2"} Nov 02 17:55:07.452680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:07.452643 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" event={"ID":"a890090c-9e21-4253-a027-05438d80f288","Type":"ContainerStarted","Data":"44b9a3089769e5c78ae07d82c3bdb2f601daa5d00709ce54bfb9903bc66f9ea5"} Nov 02 17:55:07.452680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:07.452668 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:07.454601 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:07.454570 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" event={"ID":"1103c0e1-74d5-4035-ba22-f56bd12f16dd","Type":"ContainerStarted","Data":"feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376"} Nov 02 17:55:07.454755 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:07.454624 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" event={"ID":"1103c0e1-74d5-4035-ba22-f56bd12f16dd","Type":"ContainerStarted","Data":"f8fdffa575fd154056035b1befafa4e94eb0ea839c22c111fabeca9a0570d361"} Nov 02 17:55:07.454755 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:07.454664 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:07.476614 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:07.476570 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" podStartSLOduration=1.476556948 podStartE2EDuration="1.476556948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:07.469991662 +0000 UTC m=+904.913074297" watchObservedRunningTime="2026-11-02 17:55:07.476556948 +0000 UTC m=+904.919639584" Nov 02 17:55:07.490376 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:07.490333 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" podStartSLOduration=1.4903198 podStartE2EDuration="1.4903198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:07.488814435 +0000 UTC m=+904.931897072" watchObservedRunningTime="2026-11-02 17:55:07.4903198 +0000 UTC m=+904.933402435" Nov 02 17:55:07.940795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:07.940760 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:08.053318 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.052046 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:08.145102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.143452 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:08.854197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.851265 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:08.929010 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.928974 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st"] Nov 02 17:55:08.934962 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.934938 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:08.940013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.939986 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d59fa283\"" Nov 02 17:55:08.940839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.940817 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d59fa283-dockercfg-82zpn\"" Nov 02 17:55:08.944320 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.944291 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st"] Nov 02 17:55:08.965382 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.965345 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:08.967696 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.967672 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn"] Nov 02 17:55:08.973878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.973681 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:08.987467 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:08.987439 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn"] Nov 02 17:55:09.102104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.096704 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx5m7\" (UniqueName: \"kubernetes.io/projected/7e6468e5-cb25-4e25-9f18-7778166b5bed-kube-api-access-hx5m7\") pod \"payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn\" (UID: \"7e6468e5-cb25-4e25-9f18-7778166b5bed\") " pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:09.121155 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.102771 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6468e5-cb25-4e25-9f18-7778166b5bed-plugins-config-volume\") pod \"payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn\" (UID: \"7e6468e5-cb25-4e25-9f18-7778166b5bed\") " pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:09.361020 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.360902 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee43833-c62b-4389-aab7-a95d43d43e12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st\" (UID: \"0ee43833-c62b-4389-aab7-a95d43d43e12\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:09.361285 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.361134 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct8ht\" (UniqueName: \"kubernetes.io/projected/0ee43833-c62b-4389-aab7-a95d43d43e12-kube-api-access-ct8ht\") pod \"payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st\" (UID: \"0ee43833-c62b-4389-aab7-a95d43d43e12\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:09.373594 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.373380 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx5m7\" (UniqueName: \"kubernetes.io/projected/7e6468e5-cb25-4e25-9f18-7778166b5bed-kube-api-access-hx5m7\") pod \"payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn\" (UID: \"7e6468e5-cb25-4e25-9f18-7778166b5bed\") " pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:09.380338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.380295 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6468e5-cb25-4e25-9f18-7778166b5bed-plugins-config-volume\") pod \"payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn\" (UID: \"7e6468e5-cb25-4e25-9f18-7778166b5bed\") " pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:09.403122 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.390012 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6468e5-cb25-4e25-9f18-7778166b5bed-plugins-config-volume\") pod \"payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn\" (UID: \"7e6468e5-cb25-4e25-9f18-7778166b5bed\") " pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:09.412232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.412139 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx5m7\" (UniqueName: \"kubernetes.io/projected/7e6468e5-cb25-4e25-9f18-7778166b5bed-kube-api-access-hx5m7\") pod \"payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn\" (UID: \"7e6468e5-cb25-4e25-9f18-7778166b5bed\") " pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:09.482689 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.482646 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee43833-c62b-4389-aab7-a95d43d43e12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st\" (UID: \"0ee43833-c62b-4389-aab7-a95d43d43e12\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:09.482864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.482727 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct8ht\" (UniqueName: \"kubernetes.io/projected/0ee43833-c62b-4389-aab7-a95d43d43e12-kube-api-access-ct8ht\") pod \"payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st\" (UID: \"0ee43833-c62b-4389-aab7-a95d43d43e12\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:09.483730 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.483704 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee43833-c62b-4389-aab7-a95d43d43e12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st\" (UID: \"0ee43833-c62b-4389-aab7-a95d43d43e12\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:09.493219 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.493194 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct8ht\" (UniqueName: \"kubernetes.io/projected/0ee43833-c62b-4389-aab7-a95d43d43e12-kube-api-access-ct8ht\") pod \"payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st\" (UID: \"0ee43833-c62b-4389-aab7-a95d43d43e12\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:09.552260 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.552225 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:09.594227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.594172 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:09.921855 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:09.921806 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee43833_c62b_4389_aab7_a95d43d43e12.slice/crio-6473d8553c0d93438d3affaa0515c55f57f8c106e90e29ceb94c5ee2f9d094fb WatchSource:0}: Error finding container 6473d8553c0d93438d3affaa0515c55f57f8c106e90e29ceb94c5ee2f9d094fb: Status 404 returned error can't find the container with id 6473d8553c0d93438d3affaa0515c55f57f8c106e90e29ceb94c5ee2f9d094fb Nov 02 17:55:09.923936 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.923844 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:09.927845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:09.927822 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st"] Nov 02 17:55:10.031742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:10.030760 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn"] Nov 02 17:55:10.631090 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:10.631043 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" event={"ID":"7e6468e5-cb25-4e25-9f18-7778166b5bed","Type":"ContainerStarted","Data":"7551e6d84507df41906e2a045f6eb959e74dbb413a2bccffdf3f88d49f1a58c1"} Nov 02 17:55:10.631496 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:10.631108 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" event={"ID":"7e6468e5-cb25-4e25-9f18-7778166b5bed","Type":"ContainerStarted","Data":"4a22de95215d100294e04d7e4d2bbdeabf19f8db3a5362e196c25d1f823c5db2"} Nov 02 17:55:10.631496 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:10.631140 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:10.633052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:10.633021 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" event={"ID":"0ee43833-c62b-4389-aab7-a95d43d43e12","Type":"ContainerStarted","Data":"ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325"} Nov 02 17:55:10.633221 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:10.633104 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" event={"ID":"0ee43833-c62b-4389-aab7-a95d43d43e12","Type":"ContainerStarted","Data":"6473d8553c0d93438d3affaa0515c55f57f8c106e90e29ceb94c5ee2f9d094fb"} Nov 02 17:55:10.633221 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:10.633132 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:10.650842 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:10.650790 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" podStartSLOduration=2.650772969 podStartE2EDuration="2.650772969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:10.647231877 +0000 UTC m=+908.090314512" watchObservedRunningTime="2026-11-02 17:55:10.650772969 +0000 UTC m=+908.093855609" Nov 02 17:55:10.668396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:10.668351 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" podStartSLOduration=2.6683389650000002 podStartE2EDuration="2.668338965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:10.665172246 +0000 UTC m=+908.108254879" watchObservedRunningTime="2026-11-02 17:55:10.668338965 +0000 UTC m=+908.111421591" Nov 02 17:55:10.755989 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:10.755951 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:11.397529 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:11.397479 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:12.054638 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:12.054599 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:12.171893 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:12.171773 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:12.332327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:12.332246 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:12.451434 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:12.451401 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:13.188652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.188603 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:13.332533 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.332499 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg"] Nov 02 17:55:13.335656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.335614 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:13.339072 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.339052 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:13.347518 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.347492 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg"] Nov 02 17:55:13.351504 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.351038 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-51cd4868\"" Nov 02 17:55:13.356137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.356112 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-51cd4868-dockercfg-qd9g5\"" Nov 02 17:55:13.385821 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.385781 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w"] Nov 02 17:55:13.398149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.396652 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:13.406237 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.406209 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w"] Nov 02 17:55:13.580106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.571354 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j2wb\" (UniqueName: \"kubernetes.io/projected/8e897192-ee80-4702-8eb1-ea8b2f91cc55-kube-api-access-4j2wb\") pod \"payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w\" (UID: \"8e897192-ee80-4702-8eb1-ea8b2f91cc55\") " pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:13.580106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.571420 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f812cba-ade5-4cd3-bd87-fdea5395504b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg\" (UID: \"3f812cba-ade5-4cd3-bd87-fdea5395504b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:13.580106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.571531 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e897192-ee80-4702-8eb1-ea8b2f91cc55-plugins-config-volume\") pod \"payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w\" (UID: \"8e897192-ee80-4702-8eb1-ea8b2f91cc55\") " pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:13.580106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.571680 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4qgs\" (UniqueName: \"kubernetes.io/projected/3f812cba-ade5-4cd3-bd87-fdea5395504b-kube-api-access-j4qgs\") pod \"payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg\" (UID: \"3f812cba-ade5-4cd3-bd87-fdea5395504b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:13.674714 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.674391 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j2wb\" (UniqueName: \"kubernetes.io/projected/8e897192-ee80-4702-8eb1-ea8b2f91cc55-kube-api-access-4j2wb\") pod \"payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w\" (UID: \"8e897192-ee80-4702-8eb1-ea8b2f91cc55\") " pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:13.674714 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.674463 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f812cba-ade5-4cd3-bd87-fdea5395504b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg\" (UID: \"3f812cba-ade5-4cd3-bd87-fdea5395504b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:13.674714 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.674563 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e897192-ee80-4702-8eb1-ea8b2f91cc55-plugins-config-volume\") pod \"payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w\" (UID: \"8e897192-ee80-4702-8eb1-ea8b2f91cc55\") " pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:13.674714 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.674719 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4qgs\" (UniqueName: \"kubernetes.io/projected/3f812cba-ade5-4cd3-bd87-fdea5395504b-kube-api-access-j4qgs\") pod \"payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg\" (UID: \"3f812cba-ade5-4cd3-bd87-fdea5395504b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:13.700133 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.676586 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f812cba-ade5-4cd3-bd87-fdea5395504b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg\" (UID: \"3f812cba-ade5-4cd3-bd87-fdea5395504b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:13.700133 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.685519 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e897192-ee80-4702-8eb1-ea8b2f91cc55-plugins-config-volume\") pod \"payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w\" (UID: \"8e897192-ee80-4702-8eb1-ea8b2f91cc55\") " pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:13.861853 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.861753 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j2wb\" (UniqueName: \"kubernetes.io/projected/8e897192-ee80-4702-8eb1-ea8b2f91cc55-kube-api-access-4j2wb\") pod \"payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w\" (UID: \"8e897192-ee80-4702-8eb1-ea8b2f91cc55\") " pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:13.862797 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.862767 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4qgs\" (UniqueName: \"kubernetes.io/projected/3f812cba-ade5-4cd3-bd87-fdea5395504b-kube-api-access-j4qgs\") pod \"payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg\" (UID: \"3f812cba-ade5-4cd3-bd87-fdea5395504b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:13.961671 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.961637 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:13.962659 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:13.962643 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:14.422356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:14.422310 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:14.930859 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:14.930819 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w"] Nov 02 17:55:14.942157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:14.942131 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg"] Nov 02 17:55:14.946136 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:14.946098 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e897192_ee80_4702_8eb1_ea8b2f91cc55.slice/crio-209ce6f8926a341a79ea2c7cd6c5ed12919a9cd61e62e7e379992fb5ced211e5 WatchSource:0}: Error finding container 209ce6f8926a341a79ea2c7cd6c5ed12919a9cd61e62e7e379992fb5ced211e5: Status 404 returned error can't find the container with id 209ce6f8926a341a79ea2c7cd6c5ed12919a9cd61e62e7e379992fb5ced211e5 Nov 02 17:55:15.534098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:15.529005 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:15.638785 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:15.638742 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:15.838159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:15.829990 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:16.309110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.302025 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:16.336767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.335228 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" event={"ID":"3f812cba-ade5-4cd3-bd87-fdea5395504b","Type":"ContainerStarted","Data":"5626d05de5524d2013eb11a7fae5a16499bdaf5c619ceb2cc77db8873edbc2c9"} Nov 02 17:55:16.336767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.335304 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" event={"ID":"3f812cba-ade5-4cd3-bd87-fdea5395504b","Type":"ContainerStarted","Data":"3ad0f24f9f55b3d17b0d1f43d329dbd641f451254264a415bc1a649425f15d7f"} Nov 02 17:55:16.336767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.336700 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:16.399109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.390769 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" event={"ID":"8e897192-ee80-4702-8eb1-ea8b2f91cc55","Type":"ContainerStarted","Data":"e73390ec203950e73e0a8039918b66969e8cd7ed85751a844243fbf88ba75e45"} Nov 02 17:55:16.399456 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.399435 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:16.399618 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.399598 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" event={"ID":"8e897192-ee80-4702-8eb1-ea8b2f91cc55","Type":"ContainerStarted","Data":"209ce6f8926a341a79ea2c7cd6c5ed12919a9cd61e62e7e379992fb5ced211e5"} Nov 02 17:55:16.399735 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.397430 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.420107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.411582 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" podStartSLOduration=3.411565404 podStartE2EDuration="3.411565404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:16.372133545 +0000 UTC m=+913.815216184" watchObservedRunningTime="2026-11-02 17:55:16.411565404 +0000 UTC m=+913.854648042" Nov 02 17:55:16.528102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.528018 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": context deadline exceeded" Nov 02 17:55:16.546104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.529576 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:55:16.639860 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.639744 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" podStartSLOduration=3.639724756 podStartE2EDuration="3.639724756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:16.424717256 +0000 UTC m=+913.867799896" watchObservedRunningTime="2026-11-02 17:55:16.639724756 +0000 UTC m=+914.082807393" Nov 02 17:55:16.846104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.845581 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:16.872207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.872167 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:55:16.872495 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.872470 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.892164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.887702 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:16.996881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:16.987575 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:17.020116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.004789 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.020116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.005257 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:55:17.124241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.124202 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:55:17.124527 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.124495 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.219097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.215454 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.219097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.216148 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.219097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.216515 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 02 17:55:17.219097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.217860 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.252543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.250284 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:55:17.252543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.250649 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:55:17.627159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.623311 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:17.658752 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.658719 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:55:17.892283 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.892184 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:55:17.940293 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.936315 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:55:17.975650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.975607 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:55:17.975957 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:17.975906 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.005000 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.004959 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.005547 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.005524 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:55:18.027110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.006709 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.029667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.029637 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:55:18.065115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.040174 4310 patch_prober.go:29] interesting pod/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:55:18.065115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.040228 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.292161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.248219 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 17:55:18.292161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.264202 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.301411 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.301374 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:55:18.314272 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.314236 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:55:18.405470 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.349194 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:55:18.405470 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.349767 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:55:18.405470 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.386733 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:55:18.414223 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.406566 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:55:18.414223 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.406938 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:55:18.414223 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.407012 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:55:18.613189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.612234 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:55:18.613189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.612311 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:18.660101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.659186 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:55:18.660597 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:18.660324 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:19.069220 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:19.066424 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:55:19.099103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:19.070435 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:55:19.295107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:19.272011 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.349050 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:19.348926 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.412151 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:19.412108 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.515261 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:19.515191 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.669307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:19.522411 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.778105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:19.536228 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:55:19.778105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:19.693474 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:19.778105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:19.700451 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:55:19.778105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:19.717257 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:55:19.968163 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:55:19.939009 4310 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.269s" Nov 02 17:55:20.065105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:20.062536 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:20.087297 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:20.079556 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.139680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:20.139633 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.351261 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:20.351162 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.560101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:20.555538 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:55:20.663774 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:20.663631 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.734270 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:20.734217 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:20.763230 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:20.763184 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:55:20.763874 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:20.763840 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:55:20.764325 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:20.764300 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:55:20.773206 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:20.773165 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:21.162258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:21.162210 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:21.373215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:21.373161 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:21.713110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:21.586995 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:55:21.978443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:21.673244 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 02 17:55:21.979583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:21.906472 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:55:21.979943 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:21.906517 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:21.985389 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:21.906538 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 02 17:55:21.989376 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:21.906551 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:22.121743 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:22.121703 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 17:55:22.702737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:22.258390 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:22.703236 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:22.417167 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 02 17:55:22.703539 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:22.426828 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Nov 02 17:55:22.703809 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:22.484978 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": context deadline exceeded" Nov 02 17:55:22.997264 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:22.997228 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:23.131121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.083449 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:55:23.131121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.083781 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 02 17:55:23.206054 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.206014 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-6jcls" podUID="c19cdb66-3f5e-4ac8-a77c-d0a85d130613" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Nov 02 17:55:23.226452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.043341 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:23.249162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.249065 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:23.410100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.381949 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 17:55:23.474639 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.474605 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:55:23.475023 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.475004 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:55:23.475243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.475230 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 17:55:23.475373 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.475363 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:55:23.559040 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.558951 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:23.590758 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.590724 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:55:23.592275 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.592253 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:55:23.609273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.596302 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:55:23.633614 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.633587 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:55:23.717954 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.716504 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7"] Nov 02 17:55:23.738097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.734583 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:23.768280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.759143 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:55:23.774908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.773752 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7"] Nov 02 17:55:23.855989 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.855900 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:55:23.859109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.857150 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee5218f-23ea-478e-925b-f324037224dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21dc3425-bmgq7\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:23.859109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.857379 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnt7b\" (UniqueName: \"kubernetes.io/projected/cee5218f-23ea-478e-925b-f324037224dc-kube-api-access-hnt7b\") pod \"maas-api-revoke-sub-21dc3425-bmgq7\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:23.859109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.857617 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee5218f-23ea-478e-925b-f324037224dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21dc3425-bmgq7\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:23.962847 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.962777 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee5218f-23ea-478e-925b-f324037224dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21dc3425-bmgq7\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:23.969167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.969134 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee5218f-23ea-478e-925b-f324037224dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21dc3425-bmgq7\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:23.969567 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.969544 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnt7b\" (UniqueName: \"kubernetes.io/projected/cee5218f-23ea-478e-925b-f324037224dc-kube-api-access-hnt7b\") pod \"maas-api-revoke-sub-21dc3425-bmgq7\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:23.975015 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.974327 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee5218f-23ea-478e-925b-f324037224dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21dc3425-bmgq7\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:23.975015 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:23.974954 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee5218f-23ea-478e-925b-f324037224dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21dc3425-bmgq7\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:24.020120 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.003195 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnt7b\" (UniqueName: \"kubernetes.io/projected/cee5218f-23ea-478e-925b-f324037224dc-kube-api-access-hnt7b\") pod \"maas-api-revoke-sub-21dc3425-bmgq7\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:24.106107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.106011 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:24.236419 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.236383 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:24.396486 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:24.396453 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee5218f_23ea_478e_925b_f324037224dc.slice/crio-5cfae2ef2e9b7eddd3cacc30a31ec3d36d5859f44a006975a542f52998c58ee3 WatchSource:0}: Error finding container 5cfae2ef2e9b7eddd3cacc30a31ec3d36d5859f44a006975a542f52998c58ee3: Status 404 returned error can't find the container with id 5cfae2ef2e9b7eddd3cacc30a31ec3d36d5859f44a006975a542f52998c58ee3 Nov 02 17:55:24.401228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.401204 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7"] Nov 02 17:55:24.489657 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.489621 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2"] Nov 02 17:55:24.496371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.496343 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:24.504003 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.503952 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2"] Nov 02 17:55:24.583689 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.583654 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:24.583842 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.583819 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:24.583990 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.583973 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6pg\" (UniqueName: \"kubernetes.io/projected/812f2a44-c138-4ca1-89cf-f0b4dd596322-kube-api-access-tv6pg\") pod \"maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:24.647552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.647482 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:24.685066 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.685036 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:24.685264 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.685235 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:24.685404 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.685388 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6pg\" (UniqueName: \"kubernetes.io/projected/812f2a44-c138-4ca1-89cf-f0b4dd596322-kube-api-access-tv6pg\") pod \"maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:24.686710 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.686684 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:24.688063 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.688043 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:24.697198 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.697165 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6pg\" (UniqueName: \"kubernetes.io/projected/812f2a44-c138-4ca1-89cf-f0b4dd596322-kube-api-access-tv6pg\") pod \"maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:24.818606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:24.817445 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:25.141042 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:25.141008 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" event={"ID":"cee5218f-23ea-478e-925b-f324037224dc","Type":"ContainerStarted","Data":"4cbdd708c7174dd94bdb2ca6edec9a8d0b5f0bf10b788d4543023421cd1a48b1"} Nov 02 17:55:25.141578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:25.141068 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" event={"ID":"cee5218f-23ea-478e-925b-f324037224dc","Type":"ContainerStarted","Data":"5cfae2ef2e9b7eddd3cacc30a31ec3d36d5859f44a006975a542f52998c58ee3"} Nov 02 17:55:25.146635 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:25.146603 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812f2a44_c138_4ca1_89cf_f0b4dd596322.slice/crio-ff8aa8de224ee8e5825b6f55ffde7b20bc04201710808f49c7f1e0a33bfdf727 WatchSource:0}: Error finding container ff8aa8de224ee8e5825b6f55ffde7b20bc04201710808f49c7f1e0a33bfdf727: Status 404 returned error can't find the container with id ff8aa8de224ee8e5825b6f55ffde7b20bc04201710808f49c7f1e0a33bfdf727 Nov 02 17:55:25.159495 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:25.151763 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2"] Nov 02 17:55:25.166009 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:25.165957 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" podStartSLOduration=3.165942821 podStartE2EDuration="3.165942821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:25.1592517 +0000 UTC m=+922.602334337" watchObservedRunningTime="2026-11-02 17:55:25.165942821 +0000 UTC m=+922.609025457" Nov 02 17:55:25.637220 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:25.637187 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:26.154201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.154160 4310 generic.go:356] "Generic (PLEG): container finished" podID="812f2a44-c138-4ca1-89cf-f0b4dd596322" containerID="9e2b8dd787b4e765fe3b9ae7437ca4e1424e48014563eddf8a8e26d7f4f77f36" exitCode=0 Nov 02 17:55:26.154618 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.154235 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" event={"ID":"812f2a44-c138-4ca1-89cf-f0b4dd596322","Type":"ContainerDied","Data":"9e2b8dd787b4e765fe3b9ae7437ca4e1424e48014563eddf8a8e26d7f4f77f36"} Nov 02 17:55:26.154618 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.154310 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" event={"ID":"812f2a44-c138-4ca1-89cf-f0b4dd596322","Type":"ContainerStarted","Data":"ff8aa8de224ee8e5825b6f55ffde7b20bc04201710808f49c7f1e0a33bfdf727"} Nov 02 17:55:26.156183 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.156156 4310 generic.go:356] "Generic (PLEG): container finished" podID="cee5218f-23ea-478e-925b-f324037224dc" containerID="4cbdd708c7174dd94bdb2ca6edec9a8d0b5f0bf10b788d4543023421cd1a48b1" exitCode=0 Nov 02 17:55:26.156302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.156239 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" event={"ID":"cee5218f-23ea-478e-925b-f324037224dc","Type":"ContainerDied","Data":"4cbdd708c7174dd94bdb2ca6edec9a8d0b5f0bf10b788d4543023421cd1a48b1"} Nov 02 17:55:26.572105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.563130 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6"] Nov 02 17:55:26.749166 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.744973 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:26.817388 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.813282 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6"] Nov 02 17:55:26.870238 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.870125 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:26.930431 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.927561 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:26.930431 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.927712 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:26.934099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.933800 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:26.934099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.933929 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:26.934099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.934002 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:26.934403 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.934130 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:26.934538 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:26.934478 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdjr\" (UniqueName: \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-kube-api-access-jmdjr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.057191 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.037317 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9"] Nov 02 17:55:27.073947 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.073907 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.074802 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.074780 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdjr\" (UniqueName: \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-kube-api-access-jmdjr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.075145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.075127 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.075334 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.075320 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.075554 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.075540 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.075695 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.075682 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.075813 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.075800 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.114553 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.114128 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.116206 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.115732 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9"] Nov 02 17:55:27.116206 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.115840 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:27.128028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.126802 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdjr\" (UniqueName: \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-kube-api-access-jmdjr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.128028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.127165 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.132362 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.132343 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.134007 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.133978 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.135507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.134583 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.136642 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.136377 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmfb6\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.177420 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.177382 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:27.177960 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.177527 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5ggq\" (UniqueName: \"kubernetes.io/projected/6f79c501-3e74-4391-9647-505ddc5c4129-kube-api-access-j5ggq\") pod \"maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:27.177960 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.177689 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:27.281947 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.280410 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:27.281947 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.280665 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:27.281947 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.280791 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5ggq\" (UniqueName: \"kubernetes.io/projected/6f79c501-3e74-4391-9647-505ddc5c4129-kube-api-access-j5ggq\") pod \"maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:27.282400 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.282360 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:27.287400 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.287359 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:27.290511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.290485 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5ggq\" (UniqueName: \"kubernetes.io/projected/6f79c501-3e74-4391-9647-505ddc5c4129-kube-api-access-j5ggq\") pod \"maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:27.360719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.360686 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:27.383331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.382509 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee5218f-23ea-478e-925b-f324037224dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee5218f-23ea-478e-925b-f324037224dc-maas-api-openshift-ca\") pod \"cee5218f-23ea-478e-925b-f324037224dc\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " Nov 02 17:55:27.383331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.382587 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee5218f-23ea-478e-925b-f324037224dc-kube-api-access-hnt7b\" (UniqueName: \"kubernetes.io/projected/cee5218f-23ea-478e-925b-f324037224dc-kube-api-access-hnt7b\") pod \"cee5218f-23ea-478e-925b-f324037224dc\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " Nov 02 17:55:27.383331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.382793 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee5218f-23ea-478e-925b-f324037224dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee5218f-23ea-478e-925b-f324037224dc-maas-api-xks-ca\") pod \"cee5218f-23ea-478e-925b-f324037224dc\" (UID: \"cee5218f-23ea-478e-925b-f324037224dc\") " Nov 02 17:55:27.385405 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.384470 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee5218f-23ea-478e-925b-f324037224dc-maas-api-openshift-ca" pod "cee5218f-23ea-478e-925b-f324037224dc" (UID: "cee5218f-23ea-478e-925b-f324037224dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:27.393033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.392992 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee5218f-23ea-478e-925b-f324037224dc-maas-api-xks-ca" pod "cee5218f-23ea-478e-925b-f324037224dc" (UID: "cee5218f-23ea-478e-925b-f324037224dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:27.393802 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.393773 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee5218f-23ea-478e-925b-f324037224dc-kube-api-access-hnt7b" pod "cee5218f-23ea-478e-925b-f324037224dc" (UID: "cee5218f-23ea-478e-925b-f324037224dc"). InnerVolumeSpecName "kube-api-access-hnt7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:27.406828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.406802 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:27.423043 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.423010 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:27.441227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.441201 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:27.485365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.485058 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812f2a44-c138-4ca1-89cf-f0b4dd596322-kube-api-access-tv6pg\" (UniqueName: \"kubernetes.io/projected/812f2a44-c138-4ca1-89cf-f0b4dd596322-kube-api-access-tv6pg\") pod \"812f2a44-c138-4ca1-89cf-f0b4dd596322\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " Nov 02 17:55:27.485365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.485226 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-openshift-ca\") pod \"812f2a44-c138-4ca1-89cf-f0b4dd596322\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " Nov 02 17:55:27.485919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.485623 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-xks-ca\") pod \"812f2a44-c138-4ca1-89cf-f0b4dd596322\" (UID: \"812f2a44-c138-4ca1-89cf-f0b4dd596322\") " Nov 02 17:55:27.506697 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.489925 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-openshift-ca" pod "812f2a44-c138-4ca1-89cf-f0b4dd596322" (UID: "812f2a44-c138-4ca1-89cf-f0b4dd596322"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:27.506697 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.491073 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812f2a44-c138-4ca1-89cf-f0b4dd596322-kube-api-access-tv6pg" pod "812f2a44-c138-4ca1-89cf-f0b4dd596322" (UID: "812f2a44-c138-4ca1-89cf-f0b4dd596322"). InnerVolumeSpecName "kube-api-access-tv6pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:27.506697 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.498068 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-xks-ca" pod "812f2a44-c138-4ca1-89cf-f0b4dd596322" (UID: "812f2a44-c138-4ca1-89cf-f0b4dd596322"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:27.514576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.514534 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:27.519367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.518422 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:27.519367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.519224 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee5218f-23ea-478e-925b-f324037224dc-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:27.519367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.519243 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnt7b\" (UniqueName: \"kubernetes.io/projected/cee5218f-23ea-478e-925b-f324037224dc-kube-api-access-hnt7b\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:27.519367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.519259 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812f2a44-c138-4ca1-89cf-f0b4dd596322-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:27.519367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.519280 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee5218f-23ea-478e-925b-f324037224dc-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:27.519367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.519296 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv6pg\" (UniqueName: \"kubernetes.io/projected/812f2a44-c138-4ca1-89cf-f0b4dd596322-kube-api-access-tv6pg\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:27.752814 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.752782 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6"] Nov 02 17:55:27.755053 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:27.755019 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c85a7c_1f56_4fe5_aba6_28408aaacc9c.slice/crio-74e980116b31d90054ea394d9f32c89a713b9c28efdf89753e0c88c518e24b3e WatchSource:0}: Error finding container 74e980116b31d90054ea394d9f32c89a713b9c28efdf89753e0c88c518e24b3e: Status 404 returned error can't find the container with id 74e980116b31d90054ea394d9f32c89a713b9c28efdf89753e0c88c518e24b3e Nov 02 17:55:27.757746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.757706 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:55:27.757860 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.757775 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 17:55:27.851756 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:27.851715 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f79c501_3e74_4391_9647_505ddc5c4129.slice/crio-d899ba5318b8c87b3f3cd484cbe04922c010d2434e876262f1a50a4d9b0334f3 WatchSource:0}: Error finding container d899ba5318b8c87b3f3cd484cbe04922c010d2434e876262f1a50a4d9b0334f3: Status 404 returned error can't find the container with id d899ba5318b8c87b3f3cd484cbe04922c010d2434e876262f1a50a4d9b0334f3 Nov 02 17:55:27.852126 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:27.852073 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9"] Nov 02 17:55:28.078144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.078047 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:28.353464 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.353396 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" Nov 02 17:55:28.354141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.354033 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7" event={"ID":"cee5218f-23ea-478e-925b-f324037224dc","Type":"ContainerDied","Data":"5cfae2ef2e9b7eddd3cacc30a31ec3d36d5859f44a006975a542f52998c58ee3"} Nov 02 17:55:28.354309 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.354296 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cfae2ef2e9b7eddd3cacc30a31ec3d36d5859f44a006975a542f52998c58ee3" Nov 02 17:55:28.359046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.359008 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" event={"ID":"6f79c501-3e74-4391-9647-505ddc5c4129","Type":"ContainerStarted","Data":"d899ba5318b8c87b3f3cd484cbe04922c010d2434e876262f1a50a4d9b0334f3"} Nov 02 17:55:28.361056 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.360995 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" event={"ID":"30c85a7c-1f56-4fe5-aba6-28408aaacc9c","Type":"ContainerStarted","Data":"8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c"} Nov 02 17:55:28.361056 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.361042 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" event={"ID":"30c85a7c-1f56-4fe5-aba6-28408aaacc9c","Type":"ContainerStarted","Data":"74e980116b31d90054ea394d9f32c89a713b9c28efdf89753e0c88c518e24b3e"} Nov 02 17:55:28.361640 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.361481 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:28.362829 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.362802 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" event={"ID":"812f2a44-c138-4ca1-89cf-f0b4dd596322","Type":"ContainerDied","Data":"ff8aa8de224ee8e5825b6f55ffde7b20bc04201710808f49c7f1e0a33bfdf727"} Nov 02 17:55:28.362922 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.362849 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8aa8de224ee8e5825b6f55ffde7b20bc04201710808f49c7f1e0a33bfdf727" Nov 02 17:55:28.362922 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.362912 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2" Nov 02 17:55:28.363545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.363522 4310 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nmfb6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:55:28.363629 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.363587 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" podUID="30c85a7c-1f56-4fe5-aba6-28408aaacc9c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:55:28.466793 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:28.466739 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" podStartSLOduration=2.466719739 podStartE2EDuration="2.466719739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:28.410657923 +0000 UTC m=+925.853740560" watchObservedRunningTime="2026-11-02 17:55:28.466719739 +0000 UTC m=+925.909802375" Nov 02 17:55:29.086461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:29.086429 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:29.269139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:29.265018 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw"] Nov 02 17:55:29.288702 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:29.288672 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:29.315837 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:29.315806 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:29.380645 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:29.380609 4310 generic.go:356] "Generic (PLEG): container finished" podID="6f79c501-3e74-4391-9647-505ddc5c4129" containerID="8717758ba259705e686bd00b13a9aed3317f1761b78d38ee38d92b905d9a4a15" exitCode=0 Nov 02 17:55:29.382884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:29.382844 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" event={"ID":"6f79c501-3e74-4391-9647-505ddc5c4129","Type":"ContainerDied","Data":"8717758ba259705e686bd00b13a9aed3317f1761b78d38ee38d92b905d9a4a15"} Nov 02 17:55:29.383535 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:29.383489 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 17:55:30.006694 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.006653 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:30.498229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.498189 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49"] Nov 02 17:55:30.499702 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.499675 4310 cpu_manager.go:414] "containerMap: removing container" podUID="812f2a44-c138-4ca1-89cf-f0b4dd596322" containerName="revoke-keys" Nov 02 17:55:30.499702 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.499702 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="812f2a44-c138-4ca1-89cf-f0b4dd596322" containerName="revoke-keys" Nov 02 17:55:30.499858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.499774 4310 cpu_manager.go:414] "containerMap: removing container" podUID="cee5218f-23ea-478e-925b-f324037224dc" containerName="revoke-keys" Nov 02 17:55:30.499858 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.499782 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee5218f-23ea-478e-925b-f324037224dc" containerName="revoke-keys" Nov 02 17:55:30.500174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.500153 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="812f2a44-c138-4ca1-89cf-f0b4dd596322" containerName="revoke-keys" Nov 02 17:55:30.500288 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.500191 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee5218f-23ea-478e-925b-f324037224dc" containerName="revoke-keys" Nov 02 17:55:30.506233 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.506198 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:30.512834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.512799 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49"] Nov 02 17:55:30.588208 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.588180 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:30.695197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.695071 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-xks-ca\") pod \"6f79c501-3e74-4391-9647-505ddc5c4129\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " Nov 02 17:55:30.695431 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.695413 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-openshift-ca\") pod \"6f79c501-3e74-4391-9647-505ddc5c4129\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " Nov 02 17:55:30.695512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.695450 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f79c501-3e74-4391-9647-505ddc5c4129-kube-api-access-j5ggq\" (UniqueName: \"kubernetes.io/projected/6f79c501-3e74-4391-9647-505ddc5c4129-kube-api-access-j5ggq\") pod \"6f79c501-3e74-4391-9647-505ddc5c4129\" (UID: \"6f79c501-3e74-4391-9647-505ddc5c4129\") " Nov 02 17:55:30.695785 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.695753 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-openshift-ca" pod "6f79c501-3e74-4391-9647-505ddc5c4129" (UID: "6f79c501-3e74-4391-9647-505ddc5c4129"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:30.695890 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.695871 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq9mr\" (UniqueName: \"kubernetes.io/projected/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-kube-api-access-rq9mr\") pod \"maas-api-revoke-keys-e2e-ait-d59fa283-6zw49\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:30.696218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.696197 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d59fa283-6zw49\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:30.696317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.696300 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d59fa283-6zw49\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:30.696423 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.696411 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:30.697219 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.697198 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-xks-ca" pod "6f79c501-3e74-4391-9647-505ddc5c4129" (UID: "6f79c501-3e74-4391-9647-505ddc5c4129"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:30.697706 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.697687 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f79c501-3e74-4391-9647-505ddc5c4129-kube-api-access-j5ggq" pod "6f79c501-3e74-4391-9647-505ddc5c4129" (UID: "6f79c501-3e74-4391-9647-505ddc5c4129"). InnerVolumeSpecName "kube-api-access-j5ggq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:30.797987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.797949 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d59fa283-6zw49\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:30.798228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.798040 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d59fa283-6zw49\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:30.798228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.798209 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq9mr\" (UniqueName: \"kubernetes.io/projected/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-kube-api-access-rq9mr\") pod \"maas-api-revoke-keys-e2e-ait-d59fa283-6zw49\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:30.798562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.798534 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5ggq\" (UniqueName: \"kubernetes.io/projected/6f79c501-3e74-4391-9647-505ddc5c4129-kube-api-access-j5ggq\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:30.798562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.798562 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f79c501-3e74-4391-9647-505ddc5c4129-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:30.798773 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.798754 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d59fa283-6zw49\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:30.800507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.800488 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d59fa283-6zw49\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:30.809660 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.809628 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq9mr\" (UniqueName: \"kubernetes.io/projected/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-kube-api-access-rq9mr\") pod \"maas-api-revoke-keys-e2e-ait-d59fa283-6zw49\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:30.829181 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.829144 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:30.977055 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:30.977022 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:31.173351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:31.173315 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49"] Nov 02 17:55:31.176439 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:31.176407 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599a96d5_51ba_49d6_aef7_b29e0a2dc27f.slice/crio-653dc229f590f2a1c874e678bba967f0d660eb06e181db8502dcdb36f69738d5 WatchSource:0}: Error finding container 653dc229f590f2a1c874e678bba967f0d660eb06e181db8502dcdb36f69738d5: Status 404 returned error can't find the container with id 653dc229f590f2a1c874e678bba967f0d660eb06e181db8502dcdb36f69738d5 Nov 02 17:55:31.401679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:31.401581 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" event={"ID":"599a96d5-51ba-49d6-aef7-b29e0a2dc27f","Type":"ContainerStarted","Data":"653dc229f590f2a1c874e678bba967f0d660eb06e181db8502dcdb36f69738d5"} Nov 02 17:55:31.403681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:31.403659 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" Nov 02 17:55:31.403824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:31.403711 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9" event={"ID":"6f79c501-3e74-4391-9647-505ddc5c4129","Type":"ContainerDied","Data":"d899ba5318b8c87b3f3cd484cbe04922c010d2434e876262f1a50a4d9b0334f3"} Nov 02 17:55:31.403824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:31.403766 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d899ba5318b8c87b3f3cd484cbe04922c010d2434e876262f1a50a4d9b0334f3" Nov 02 17:55:32.244300 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:32.244267 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:32.393130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:32.393072 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:32.437031 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:32.436292 4310 generic.go:356] "Generic (PLEG): container finished" podID="599a96d5-51ba-49d6-aef7-b29e0a2dc27f" containerID="b5d97794db6cb57aa9d020a0167bf2b4d516688c9faddf1f3b320fe1b9638e63" exitCode=0 Nov 02 17:55:32.437031 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:32.436574 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" event={"ID":"599a96d5-51ba-49d6-aef7-b29e0a2dc27f","Type":"ContainerDied","Data":"b5d97794db6cb57aa9d020a0167bf2b4d516688c9faddf1f3b320fe1b9638e63"} Nov 02 17:55:33.079418 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.079380 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:33.529863 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:55:33.529802 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:55:33.628604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.628570 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:33.697720 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.697681 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-xks-ca\") pod \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " Nov 02 17:55:33.697905 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.697853 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-kube-api-access-rq9mr\" (UniqueName: \"kubernetes.io/projected/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-kube-api-access-rq9mr\") pod \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " Nov 02 17:55:33.697905 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.697883 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-openshift-ca\") pod \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\" (UID: \"599a96d5-51ba-49d6-aef7-b29e0a2dc27f\") " Nov 02 17:55:33.699199 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.698386 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-openshift-ca" pod "599a96d5-51ba-49d6-aef7-b29e0a2dc27f" (UID: "599a96d5-51ba-49d6-aef7-b29e0a2dc27f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:33.699199 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.698754 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:33.700370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.700348 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-xks-ca" pod "599a96d5-51ba-49d6-aef7-b29e0a2dc27f" (UID: "599a96d5-51ba-49d6-aef7-b29e0a2dc27f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:33.701251 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.701223 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-kube-api-access-rq9mr" pod "599a96d5-51ba-49d6-aef7-b29e0a2dc27f" (UID: "599a96d5-51ba-49d6-aef7-b29e0a2dc27f"). InnerVolumeSpecName "kube-api-access-rq9mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:33.801911 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.801881 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq9mr\" (UniqueName: \"kubernetes.io/projected/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-kube-api-access-rq9mr\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:33.801911 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.801908 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/599a96d5-51ba-49d6-aef7-b29e0a2dc27f-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:33.827297 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:33.827254 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:34.460469 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.460444 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" Nov 02 17:55:34.460618 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.460487 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49" event={"ID":"599a96d5-51ba-49d6-aef7-b29e0a2dc27f","Type":"ContainerDied","Data":"653dc229f590f2a1c874e678bba967f0d660eb06e181db8502dcdb36f69738d5"} Nov 02 17:55:34.460618 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.460536 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="653dc229f590f2a1c874e678bba967f0d660eb06e181db8502dcdb36f69738d5" Nov 02 17:55:34.564438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.564400 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9"] Nov 02 17:55:34.581812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.581558 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh"] Nov 02 17:55:34.623379 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.623340 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:34.657588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.657557 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b"] Nov 02 17:55:34.658772 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.658746 4310 cpu_manager.go:414] "containerMap: removing container" podUID="6f79c501-3e74-4391-9647-505ddc5c4129" containerName="revoke-keys" Nov 02 17:55:34.658772 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.658768 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f79c501-3e74-4391-9647-505ddc5c4129" containerName="revoke-keys" Nov 02 17:55:34.658943 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.658848 4310 cpu_manager.go:414] "containerMap: removing container" podUID="599a96d5-51ba-49d6-aef7-b29e0a2dc27f" containerName="revoke-keys" Nov 02 17:55:34.658943 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.658857 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599a96d5-51ba-49d6-aef7-b29e0a2dc27f" containerName="revoke-keys" Nov 02 17:55:34.659948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.659095 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="599a96d5-51ba-49d6-aef7-b29e0a2dc27f" containerName="revoke-keys" Nov 02 17:55:34.659948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.659116 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f79c501-3e74-4391-9647-505ddc5c4129" containerName="revoke-keys" Nov 02 17:55:34.677093 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.671344 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:34.679603 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.679575 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:55:34.679735 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.679720 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:55:34.719187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.719151 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:34.723402 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.723367 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2686d\" (UniqueName: \"kubernetes.io/projected/6b603fc8-c049-4a35-a1ad-9dbd487a2947-kube-api-access-2686d\") pod \"maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:34.723580 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.723444 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:34.723876 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.723855 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:34.728764 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.728736 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b"] Nov 02 17:55:34.825452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.825421 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2686d\" (UniqueName: \"kubernetes.io/projected/6b603fc8-c049-4a35-a1ad-9dbd487a2947-kube-api-access-2686d\") pod \"maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:34.825603 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.825468 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:34.825746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.825732 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:34.826411 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.826387 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:34.827966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.827940 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:34.833839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:34.833818 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2686d\" (UniqueName: \"kubernetes.io/projected/6b603fc8-c049-4a35-a1ad-9dbd487a2947-kube-api-access-2686d\") pod \"maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:35.030443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:35.030355 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:35.555690 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:35.555660 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b603fc8_c049_4a35_a1ad_9dbd487a2947.slice/crio-d86013c42ca0e6ae8656fab846c3ce9d5b1fb8d1c85f43e6051a7cf813b6b9c7 WatchSource:0}: Error finding container d86013c42ca0e6ae8656fab846c3ce9d5b1fb8d1c85f43e6051a7cf813b6b9c7: Status 404 returned error can't find the container with id d86013c42ca0e6ae8656fab846c3ce9d5b1fb8d1c85f43e6051a7cf813b6b9c7 Nov 02 17:55:35.558859 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:35.558832 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b"] Nov 02 17:55:35.843031 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:35.842911 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:36.481694 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:36.481663 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" event={"ID":"6b603fc8-c049-4a35-a1ad-9dbd487a2947","Type":"ContainerStarted","Data":"c4c6617d7e330dc9bdfde2d07b7ca48a3e0a26f97d0cb1e2d2337bf6cf4122aa"} Nov 02 17:55:36.482261 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:36.481711 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" event={"ID":"6b603fc8-c049-4a35-a1ad-9dbd487a2947","Type":"ContainerStarted","Data":"d86013c42ca0e6ae8656fab846c3ce9d5b1fb8d1c85f43e6051a7cf813b6b9c7"} Nov 02 17:55:36.504358 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:36.504303 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" podStartSLOduration=2.504288312 podStartE2EDuration="2.504288312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:36.497013215 +0000 UTC m=+933.940095850" watchObservedRunningTime="2026-11-02 17:55:36.504288312 +0000 UTC m=+933.947370948" Nov 02 17:55:36.755286 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:36.755200 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:37.070257 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.070177 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h"] Nov 02 17:55:37.082269 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.082242 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8"] Nov 02 17:55:37.129623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.129586 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:37.231869 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.231828 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:37.548232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.545796 4310 generic.go:356] "Generic (PLEG): container finished" podID="6b603fc8-c049-4a35-a1ad-9dbd487a2947" containerID="c4c6617d7e330dc9bdfde2d07b7ca48a3e0a26f97d0cb1e2d2337bf6cf4122aa" exitCode=0 Nov 02 17:55:37.548232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.546133 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" event={"ID":"6b603fc8-c049-4a35-a1ad-9dbd487a2947","Type":"ContainerDied","Data":"c4c6617d7e330dc9bdfde2d07b7ca48a3e0a26f97d0cb1e2d2337bf6cf4122aa"} Nov 02 17:55:37.730544 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.730512 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6"] Nov 02 17:55:37.792039 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.789456 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n"] Nov 02 17:55:37.792039 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.790948 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:37.797751 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.797723 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:37.799605 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.799476 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6"] Nov 02 17:55:37.822586 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.811292 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n"] Nov 02 17:55:37.874806 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.874448 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrft\" (UniqueName: \"kubernetes.io/projected/69353ed1-5667-4407-a4bc-36105ebcb410-kube-api-access-rqrft\") pod \"maas-api-revoke-sub-719e5d3e-7bkd6\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:37.874806 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.874612 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwhhb\" (UniqueName: \"kubernetes.io/projected/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-kube-api-access-gwhhb\") pod \"maas-api-revoke-sub-8bdcf90d-ghl5n\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:37.874806 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.874731 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719e5d3e-7bkd6\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:37.875044 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.874889 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bdcf90d-ghl5n\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:37.875139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.875118 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bdcf90d-ghl5n\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:37.875234 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.875187 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719e5d3e-7bkd6\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:37.976455 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.976411 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719e5d3e-7bkd6\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:37.976653 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.976578 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrft\" (UniqueName: \"kubernetes.io/projected/69353ed1-5667-4407-a4bc-36105ebcb410-kube-api-access-rqrft\") pod \"maas-api-revoke-sub-719e5d3e-7bkd6\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:37.976653 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.976618 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwhhb\" (UniqueName: \"kubernetes.io/projected/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-kube-api-access-gwhhb\") pod \"maas-api-revoke-sub-8bdcf90d-ghl5n\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:37.976784 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.976660 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719e5d3e-7bkd6\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:37.976784 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.976719 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bdcf90d-ghl5n\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:37.976951 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.976934 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bdcf90d-ghl5n\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:37.977220 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.977199 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-719e5d3e-7bkd6\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:37.977625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.977602 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bdcf90d-ghl5n\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:37.979176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.979149 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bdcf90d-ghl5n\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:37.979303 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.979284 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-719e5d3e-7bkd6\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:37.984966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.984944 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrft\" (UniqueName: \"kubernetes.io/projected/69353ed1-5667-4407-a4bc-36105ebcb410-kube-api-access-rqrft\") pod \"maas-api-revoke-sub-719e5d3e-7bkd6\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:37.985831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.985814 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwhhb\" (UniqueName: \"kubernetes.io/projected/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-kube-api-access-gwhhb\") pod \"maas-api-revoke-sub-8bdcf90d-ghl5n\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:37.987868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:37.987844 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:38.117276 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.111884 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:38.124160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.123356 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:38.501812 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:38.501780 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ef6ff4_a23e_45b0_900e_3c3c057df7cd.slice/crio-9f08697a3643889f6f61a94a398e3903e526f642ae50fb5d786e3480f7e131ae WatchSource:0}: Error finding container 9f08697a3643889f6f61a94a398e3903e526f642ae50fb5d786e3480f7e131ae: Status 404 returned error can't find the container with id 9f08697a3643889f6f61a94a398e3903e526f642ae50fb5d786e3480f7e131ae Nov 02 17:55:38.505407 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.505388 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n"] Nov 02 17:55:38.505759 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:55:38.505738 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69353ed1_5667_4407_a4bc_36105ebcb410.slice/crio-a189a3b4f116cc374c338ec2ab33de6bf9aaf1a70c3d43bda1a2b9f8d5576e03 WatchSource:0}: Error finding container a189a3b4f116cc374c338ec2ab33de6bf9aaf1a70c3d43bda1a2b9f8d5576e03: Status 404 returned error can't find the container with id a189a3b4f116cc374c338ec2ab33de6bf9aaf1a70c3d43bda1a2b9f8d5576e03 Nov 02 17:55:38.514054 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.514030 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6"] Nov 02 17:55:38.557020 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.556995 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" event={"ID":"69353ed1-5667-4407-a4bc-36105ebcb410","Type":"ContainerStarted","Data":"a189a3b4f116cc374c338ec2ab33de6bf9aaf1a70c3d43bda1a2b9f8d5576e03"} Nov 02 17:55:38.558687 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.558660 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" event={"ID":"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd","Type":"ContainerStarted","Data":"9f08697a3643889f6f61a94a398e3903e526f642ae50fb5d786e3480f7e131ae"} Nov 02 17:55:38.758281 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.758200 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:38.796859 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.796824 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:38.800331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.800306 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-openshift-ca\") pod \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " Nov 02 17:55:38.800547 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.800531 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b603fc8-c049-4a35-a1ad-9dbd487a2947-kube-api-access-2686d\" (UniqueName: \"kubernetes.io/projected/6b603fc8-c049-4a35-a1ad-9dbd487a2947-kube-api-access-2686d\") pod \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " Nov 02 17:55:38.800656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.800642 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-xks-ca\") pod \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\" (UID: \"6b603fc8-c049-4a35-a1ad-9dbd487a2947\") " Nov 02 17:55:38.800763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.800730 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-openshift-ca" pod "6b603fc8-c049-4a35-a1ad-9dbd487a2947" (UID: "6b603fc8-c049-4a35-a1ad-9dbd487a2947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:38.801572 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.801451 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.802884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.802855 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b603fc8-c049-4a35-a1ad-9dbd487a2947-kube-api-access-2686d" pod "6b603fc8-c049-4a35-a1ad-9dbd487a2947" (UID: "6b603fc8-c049-4a35-a1ad-9dbd487a2947"). InnerVolumeSpecName "kube-api-access-2686d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:38.802884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.802875 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-xks-ca" pod "6b603fc8-c049-4a35-a1ad-9dbd487a2947" (UID: "6b603fc8-c049-4a35-a1ad-9dbd487a2947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:38.904587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.904010 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2686d\" (UniqueName: \"kubernetes.io/projected/6b603fc8-c049-4a35-a1ad-9dbd487a2947-kube-api-access-2686d\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.904587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:38.904044 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b603fc8-c049-4a35-a1ad-9dbd487a2947-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:39.088533 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:39.088499 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:39.484707 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:39.484665 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:39.566105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:39.565160 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" podUID="b66cbc84-3c7c-471b-914a-855635642fc8" containerName="payload-processing" containerID="cri-o://3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af" gracePeriod=25 Nov 02 17:55:39.621130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:39.610523 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" podUID="c467c116-1975-4e66-ab08-42ec962e170f" containerName="payload-pre-processing" containerID="cri-o://5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003" gracePeriod=25 Nov 02 17:55:39.646151 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:39.642779 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" event={"ID":"69353ed1-5667-4407-a4bc-36105ebcb410","Type":"ContainerStarted","Data":"204e3a6e88fc30f355337c50c815ce610ac3917010844d336ba606d38611cbce"} Nov 02 17:55:39.647844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:39.646650 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" event={"ID":"6b603fc8-c049-4a35-a1ad-9dbd487a2947","Type":"ContainerDied","Data":"d86013c42ca0e6ae8656fab846c3ce9d5b1fb8d1c85f43e6051a7cf813b6b9c7"} Nov 02 17:55:39.647844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:39.646722 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d86013c42ca0e6ae8656fab846c3ce9d5b1fb8d1c85f43e6051a7cf813b6b9c7" Nov 02 17:55:39.647844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:39.646810 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b" Nov 02 17:55:39.704128 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:39.703457 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" event={"ID":"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd","Type":"ContainerStarted","Data":"3d34b8dc514fd0089e09a79cb54494c24420fc573263a9853b99bba1bdfd05ca"} Nov 02 17:55:39.713431 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:39.713377 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" podStartSLOduration=2.713360342 podStartE2EDuration="2.713360342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:39.711330277 +0000 UTC m=+937.154412940" watchObservedRunningTime="2026-11-02 17:55:39.713360342 +0000 UTC m=+937.156443009" Nov 02 17:55:39.740877 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:39.740782 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" podStartSLOduration=2.7407689250000002 podStartE2EDuration="2.740768925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:39.734898927 +0000 UTC m=+937.177981564" watchObservedRunningTime="2026-11-02 17:55:39.740768925 +0000 UTC m=+937.183851560" Nov 02 17:55:40.182287 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:40.182210 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:40.611350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:40.611319 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn"] Nov 02 17:55:40.627673 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:40.627638 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st"] Nov 02 17:55:40.719467 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:40.719429 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:40.731378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:40.731337 4310 generic.go:356] "Generic (PLEG): container finished" podID="69353ed1-5667-4407-a4bc-36105ebcb410" containerID="204e3a6e88fc30f355337c50c815ce610ac3917010844d336ba606d38611cbce" exitCode=0 Nov 02 17:55:40.731606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:40.731588 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" event={"ID":"69353ed1-5667-4407-a4bc-36105ebcb410","Type":"ContainerDied","Data":"204e3a6e88fc30f355337c50c815ce610ac3917010844d336ba606d38611cbce"} Nov 02 17:55:40.734548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:40.734517 4310 generic.go:356] "Generic (PLEG): container finished" podID="c3ef6ff4-a23e-45b0-900e-3c3c057df7cd" containerID="3d34b8dc514fd0089e09a79cb54494c24420fc573263a9853b99bba1bdfd05ca" exitCode=0 Nov 02 17:55:40.734695 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:40.734561 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" event={"ID":"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd","Type":"ContainerDied","Data":"3d34b8dc514fd0089e09a79cb54494c24420fc573263a9853b99bba1bdfd05ca"} Nov 02 17:55:40.809540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:40.809506 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:41.011109 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:41.007668 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:41.248103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:41.243399 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" podUID="b66cbc84-3c7c-471b-914a-855635642fc8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:55:41.277453 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:41.277374 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" podUID="c467c116-1975-4e66-ab08-42ec962e170f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:55:41.882617 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:41.882580 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:42.085793 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.071011 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" podUID="1103c0e1-74d5-4035-ba22-f56bd12f16dd" containerName="payload-processing" containerID="cri-o://feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376" gracePeriod=25 Nov 02 17:55:42.086387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.086331 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" podUID="a890090c-9e21-4253-a027-05438d80f288" containerName="payload-pre-processing" containerID="cri-o://c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2" gracePeriod=25 Nov 02 17:55:42.169671 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.169643 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:42.232127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.231224 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:42.262045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.260390 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-xks-ca\") pod \"69353ed1-5667-4407-a4bc-36105ebcb410\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " Nov 02 17:55:42.262045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.260606 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-openshift-ca\") pod \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " Nov 02 17:55:42.262045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.260656 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-kube-api-access-gwhhb\" (UniqueName: \"kubernetes.io/projected/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-kube-api-access-gwhhb\") pod \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " Nov 02 17:55:42.262045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.260749 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-xks-ca\") pod \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\" (UID: \"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd\") " Nov 02 17:55:42.262045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.260831 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69353ed1-5667-4407-a4bc-36105ebcb410-kube-api-access-rqrft\" (UniqueName: \"kubernetes.io/projected/69353ed1-5667-4407-a4bc-36105ebcb410-kube-api-access-rqrft\") pod \"69353ed1-5667-4407-a4bc-36105ebcb410\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " Nov 02 17:55:42.262045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.260903 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-openshift-ca\") pod \"69353ed1-5667-4407-a4bc-36105ebcb410\" (UID: \"69353ed1-5667-4407-a4bc-36105ebcb410\") " Nov 02 17:55:42.262424 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.262359 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-openshift-ca" pod "69353ed1-5667-4407-a4bc-36105ebcb410" (UID: "69353ed1-5667-4407-a4bc-36105ebcb410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:42.263001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.262957 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-openshift-ca" pod "c3ef6ff4-a23e-45b0-900e-3c3c057df7cd" (UID: "c3ef6ff4-a23e-45b0-900e-3c3c057df7cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:42.267158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.266532 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-xks-ca" pod "c3ef6ff4-a23e-45b0-900e-3c3c057df7cd" (UID: "c3ef6ff4-a23e-45b0-900e-3c3c057df7cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:42.267158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.266598 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69353ed1-5667-4407-a4bc-36105ebcb410-kube-api-access-rqrft" pod "69353ed1-5667-4407-a4bc-36105ebcb410" (UID: "69353ed1-5667-4407-a4bc-36105ebcb410"). InnerVolumeSpecName "kube-api-access-rqrft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:42.267980 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.267958 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-xks-ca" pod "69353ed1-5667-4407-a4bc-36105ebcb410" (UID: "69353ed1-5667-4407-a4bc-36105ebcb410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:42.268255 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.268229 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-kube-api-access-gwhhb" pod "c3ef6ff4-a23e-45b0-900e-3c3c057df7cd" (UID: "c3ef6ff4-a23e-45b0-900e-3c3c057df7cd"). InnerVolumeSpecName "kube-api-access-gwhhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:42.364399 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.364362 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.364561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.364404 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqrft\" (UniqueName: \"kubernetes.io/projected/69353ed1-5667-4407-a4bc-36105ebcb410-kube-api-access-rqrft\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.364561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.364420 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.364561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.364434 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69353ed1-5667-4407-a4bc-36105ebcb410-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.364561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.364447 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.364561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.364459 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwhhb\" (UniqueName: \"kubernetes.io/projected/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd-kube-api-access-gwhhb\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.396672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.396634 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:42.447153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.447125 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:42.466158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.466130 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1103c0e1-74d5-4035-ba22-f56bd12f16dd-kube-api-access-2jpst\" (UniqueName: \"kubernetes.io/projected/1103c0e1-74d5-4035-ba22-f56bd12f16dd-kube-api-access-2jpst\") pod \"1103c0e1-74d5-4035-ba22-f56bd12f16dd\" (UID: \"1103c0e1-74d5-4035-ba22-f56bd12f16dd\") " Nov 02 17:55:42.466403 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.466382 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a890090c-9e21-4253-a027-05438d80f288-kube-api-access-7fbpw\" (UniqueName: \"kubernetes.io/projected/a890090c-9e21-4253-a027-05438d80f288-kube-api-access-7fbpw\") pod \"a890090c-9e21-4253-a027-05438d80f288\" (UID: \"a890090c-9e21-4253-a027-05438d80f288\") " Nov 02 17:55:42.466602 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.466584 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a890090c-9e21-4253-a027-05438d80f288-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a890090c-9e21-4253-a027-05438d80f288-plugins-config-volume\") pod \"a890090c-9e21-4253-a027-05438d80f288\" (UID: \"a890090c-9e21-4253-a027-05438d80f288\") " Nov 02 17:55:42.466701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.466683 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1103c0e1-74d5-4035-ba22-f56bd12f16dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1103c0e1-74d5-4035-ba22-f56bd12f16dd-plugins-config-volume\") pod \"1103c0e1-74d5-4035-ba22-f56bd12f16dd\" (UID: \"1103c0e1-74d5-4035-ba22-f56bd12f16dd\") " Nov 02 17:55:42.467196 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.467166 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a890090c-9e21-4253-a027-05438d80f288-plugins-config-volume" pod "a890090c-9e21-4253-a027-05438d80f288" (UID: "a890090c-9e21-4253-a027-05438d80f288"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:42.467398 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.467376 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1103c0e1-74d5-4035-ba22-f56bd12f16dd-plugins-config-volume" pod "1103c0e1-74d5-4035-ba22-f56bd12f16dd" (UID: "1103c0e1-74d5-4035-ba22-f56bd12f16dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:42.467566 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.467545 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a890090c-9e21-4253-a027-05438d80f288-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.467608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.467575 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1103c0e1-74d5-4035-ba22-f56bd12f16dd-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.470666 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.470315 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a890090c-9e21-4253-a027-05438d80f288-kube-api-access-7fbpw" pod "a890090c-9e21-4253-a027-05438d80f288" (UID: "a890090c-9e21-4253-a027-05438d80f288"). InnerVolumeSpecName "kube-api-access-7fbpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:42.471009 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.470987 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1103c0e1-74d5-4035-ba22-f56bd12f16dd-kube-api-access-2jpst" pod "1103c0e1-74d5-4035-ba22-f56bd12f16dd" (UID: "1103c0e1-74d5-4035-ba22-f56bd12f16dd"). InnerVolumeSpecName "kube-api-access-2jpst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:42.538815 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.538775 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:42.569741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.569701 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fbpw\" (UniqueName: \"kubernetes.io/projected/a890090c-9e21-4253-a027-05438d80f288-kube-api-access-7fbpw\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.569741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.569729 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jpst\" (UniqueName: \"kubernetes.io/projected/1103c0e1-74d5-4035-ba22-f56bd12f16dd-kube-api-access-2jpst\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.778149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.774927 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:42.859461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.859426 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9"] Nov 02 17:55:42.870687 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.868900 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:42.877177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.876837 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-be1b0f58-4hgm9"] Nov 02 17:55:42.925101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.914625 4310 generic.go:356] "Generic (PLEG): container finished" podID="1103c0e1-74d5-4035-ba22-f56bd12f16dd" containerID="feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376" exitCode=0 Nov 02 17:55:42.925101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.914761 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" event={"ID":"1103c0e1-74d5-4035-ba22-f56bd12f16dd","Type":"ContainerDied","Data":"feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376"} Nov 02 17:55:42.925101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.914815 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" event={"ID":"1103c0e1-74d5-4035-ba22-f56bd12f16dd","Type":"ContainerDied","Data":"f8fdffa575fd154056035b1befafa4e94eb0ea839c22c111fabeca9a0570d361"} Nov 02 17:55:42.925101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.914847 4310 scope.go:122] "RemoveContainer" containerID="feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376" Nov 02 17:55:42.925101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.915015 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h" Nov 02 17:55:42.925101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.920156 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" event={"ID":"69353ed1-5667-4407-a4bc-36105ebcb410","Type":"ContainerDied","Data":"a189a3b4f116cc374c338ec2ab33de6bf9aaf1a70c3d43bda1a2b9f8d5576e03"} Nov 02 17:55:42.925101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.920241 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a189a3b4f116cc374c338ec2ab33de6bf9aaf1a70c3d43bda1a2b9f8d5576e03" Nov 02 17:55:42.925101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.920325 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6" Nov 02 17:55:42.997101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:42.991799 4310 scope.go:122] "RemoveContainer" containerID="feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376" Nov 02 17:55:43.012127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.001587 4310 generic.go:356] "Generic (PLEG): container finished" podID="a890090c-9e21-4253-a027-05438d80f288" containerID="c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2" exitCode=0 Nov 02 17:55:43.012127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.001763 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" event={"ID":"a890090c-9e21-4253-a027-05438d80f288","Type":"ContainerDied","Data":"c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2"} Nov 02 17:55:43.012127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.001812 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" event={"ID":"a890090c-9e21-4253-a027-05438d80f288","Type":"ContainerDied","Data":"44b9a3089769e5c78ae07d82c3bdb2f601daa5d00709ce54bfb9903bc66f9ea5"} Nov 02 17:55:43.012127 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:55:43.001877 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376\": container with ID starting with feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376 not found: ID does not exist" containerID="feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376" Nov 02 17:55:43.012127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.001907 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376"} err="failed to get container status \"feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376\": rpc error: code = NotFound desc = could not find container \"feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376\": container with ID starting with feaf7988cc605258520a8c3d7f9398aadc11d2593fe2d43ab98b6ea4c4415376 not found: ID does not exist" Nov 02 17:55:43.012127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.001928 4310 scope.go:122] "RemoveContainer" containerID="c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2" Nov 02 17:55:43.012127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.002266 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8" Nov 02 17:55:43.012127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.004961 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:43.019284 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.017278 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" event={"ID":"c3ef6ff4-a23e-45b0-900e-3c3c057df7cd","Type":"ContainerDied","Data":"9f08697a3643889f6f61a94a398e3903e526f642ae50fb5d786e3480f7e131ae"} Nov 02 17:55:43.019284 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.017327 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f08697a3643889f6f61a94a398e3903e526f642ae50fb5d786e3480f7e131ae" Nov 02 17:55:43.019284 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.019145 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n" Nov 02 17:55:43.035523 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.035501 4310 scope.go:122] "RemoveContainer" containerID="c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2" Nov 02 17:55:43.036545 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:55:43.035830 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2\": container with ID starting with c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2 not found: ID does not exist" containerID="c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2" Nov 02 17:55:43.036545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.035862 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2"} err="failed to get container status \"c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2\": rpc error: code = NotFound desc = could not find container \"c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2\": container with ID starting with c332963c536fd25f36157e49a61f5f2b00cca5f03d5bf3baa829d6e7177cd3c2 not found: ID does not exist" Nov 02 17:55:43.050058 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.050034 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h"] Nov 02 17:55:43.059190 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.059165 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-be1b0f58-54ffb94b67-2xw4h"] Nov 02 17:55:43.065719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.065696 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8"] Nov 02 17:55:43.072063 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.072039 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-be1b0f58-7f98847b7f-dnjr8"] Nov 02 17:55:43.109220 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.109167 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:43.623511 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.623465 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:43.990372 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:43.990341 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:44.244718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.244624 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:44.343337 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.343293 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" containerID="cri-o://cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a" gracePeriod=45 Nov 02 17:55:44.596327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.596294 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/ready\": dial tcp 10.133.0.102:8000: connect: connection refused" Nov 02 17:55:44.621777 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.621733 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:44.698454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.698416 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1103c0e1-74d5-4035-ba22-f56bd12f16dd" path="/var/lib/kubelet/pods/1103c0e1-74d5-4035-ba22-f56bd12f16dd/volumes" Nov 02 17:55:44.699434 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.699409 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f79c501-3e74-4391-9647-505ddc5c4129" path="/var/lib/kubelet/pods/6f79c501-3e74-4391-9647-505ddc5c4129/volumes" Nov 02 17:55:44.700179 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.700152 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a890090c-9e21-4253-a027-05438d80f288" path="/var/lib/kubelet/pods/a890090c-9e21-4253-a027-05438d80f288/volumes" Nov 02 17:55:44.746110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.746066 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:55:44.806722 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.806608 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w"] Nov 02 17:55:44.821256 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.821223 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg"] Nov 02 17:55:44.861911 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.861879 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-tmp-dir\") pod \"29d27280-8d99-4708-be3a-e4781a0c43b0\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " Nov 02 17:55:44.862135 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.862012 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29d27280-8d99-4708-be3a-e4781a0c43b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/29d27280-8d99-4708-be3a-e4781a0c43b0-tls-certs\") pod \"29d27280-8d99-4708-be3a-e4781a0c43b0\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " Nov 02 17:55:44.862411 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.862371 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-tmp-dir" pod "29d27280-8d99-4708-be3a-e4781a0c43b0" (UID: "29d27280-8d99-4708-be3a-e4781a0c43b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.863033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.862696 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-dshm\") pod \"29d27280-8d99-4708-be3a-e4781a0c43b0\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " Nov 02 17:55:44.863033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.862864 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-model-cache\") pod \"29d27280-8d99-4708-be3a-e4781a0c43b0\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " Nov 02 17:55:44.863033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.862916 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-home\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-home\") pod \"29d27280-8d99-4708-be3a-e4781a0c43b0\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " Nov 02 17:55:44.863033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.862957 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d27280-8d99-4708-be3a-e4781a0c43b0-kube-api-access-wzccq\" (UniqueName: \"kubernetes.io/projected/29d27280-8d99-4708-be3a-e4781a0c43b0-kube-api-access-wzccq\") pod \"29d27280-8d99-4708-be3a-e4781a0c43b0\" (UID: \"29d27280-8d99-4708-be3a-e4781a0c43b0\") " Nov 02 17:55:44.863383 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.863362 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-model-cache" pod "29d27280-8d99-4708-be3a-e4781a0c43b0" (UID: "29d27280-8d99-4708-be3a-e4781a0c43b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.863727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.863706 4310 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-model-cache\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.863727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.863727 4310 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-tmp-dir\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.866513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.866489 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-home" pod "29d27280-8d99-4708-be3a-e4781a0c43b0" (UID: "29d27280-8d99-4708-be3a-e4781a0c43b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.866642 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.866597 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-dshm" pod "29d27280-8d99-4708-be3a-e4781a0c43b0" (UID: "29d27280-8d99-4708-be3a-e4781a0c43b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:44.867650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.867621 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d27280-8d99-4708-be3a-e4781a0c43b0-kube-api-access-wzccq" pod "29d27280-8d99-4708-be3a-e4781a0c43b0" (UID: "29d27280-8d99-4708-be3a-e4781a0c43b0"). InnerVolumeSpecName "kube-api-access-wzccq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:44.868414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.868394 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29d27280-8d99-4708-be3a-e4781a0c43b0-tls-certs" pod "29d27280-8d99-4708-be3a-e4781a0c43b0" (UID: "29d27280-8d99-4708-be3a-e4781a0c43b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:44.880779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.880748 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:44.964868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.964831 4310 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d27280-8d99-4708-be3a-e4781a0c43b0-tls-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.964868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.964866 4310 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-dshm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.964868 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.964879 4310 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d27280-8d99-4708-be3a-e4781a0c43b0-home\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.965224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.964891 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzccq\" (UniqueName: \"kubernetes.io/projected/29d27280-8d99-4708-be3a-e4781a0c43b0-kube-api-access-wzccq\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:44.977889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:44.977859 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:45.044886 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.044851 4310 generic.go:356] "Generic (PLEG): container finished" podID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerID="cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a" exitCode=0 Nov 02 17:55:45.045369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.044897 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" Nov 02 17:55:45.045369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.044935 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" event={"ID":"29d27280-8d99-4708-be3a-e4781a0c43b0","Type":"ContainerDied","Data":"cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a"} Nov 02 17:55:45.045369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.044978 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw" event={"ID":"29d27280-8d99-4708-be3a-e4781a0c43b0","Type":"ContainerDied","Data":"a1aed7690944c5662e89408582f7328c01b6434c54787356a6d34f5e0581bca6"} Nov 02 17:55:45.045369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.044997 4310 scope.go:122] "RemoveContainer" containerID="cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a" Nov 02 17:55:45.060742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.060719 4310 scope.go:122] "RemoveContainer" containerID="cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a" Nov 02 17:55:45.061093 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:55:45.061058 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a\": container with ID starting with cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a not found: ID does not exist" containerID="cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a" Nov 02 17:55:45.061230 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.061102 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a"} err="failed to get container status \"cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a\": rpc error: code = NotFound desc = could not find container \"cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a\": container with ID starting with cb32d47a7346defc3ef81db7989ef1d40baca600c69dfa90613e55780155673a not found: ID does not exist" Nov 02 17:55:45.071591 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.071563 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:45.077592 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.077565 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw"] Nov 02 17:55:45.084290 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.084269 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7b0683/distinct-2-w6-7b0683-kserve-7866445f8-ntfdw"] Nov 02 17:55:45.161565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.161528 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:45.348173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.347964 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:45.511155 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.508156 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49"] Nov 02 17:55:45.529106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.515688 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:45.545020 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.544978 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d59fa283-6zw49"] Nov 02 17:55:45.618796 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.616605 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" podUID="7e6468e5-cb25-4e25-9f18-7778166b5bed" containerName="payload-processing" containerID="cri-o://7551e6d84507df41906e2a045f6eb959e74dbb413a2bccffdf3f88d49f1a58c1" gracePeriod=25 Nov 02 17:55:45.637296 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.632323 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" podUID="0ee43833-c62b-4389-aab7-a95d43d43e12" containerName="payload-pre-processing" containerID="cri-o://ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325" gracePeriod=25 Nov 02 17:55:45.816025 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.815991 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:45.844099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.840988 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk"] Nov 02 17:55:45.844099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:45.841309 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" podUID="6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" containerName="istio-proxy" containerID="cri-o://2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c" gracePeriod=30 Nov 02 17:55:46.360348 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.359878 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:46.367630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.367606 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:46.377038 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.377002 4310 generic.go:356] "Generic (PLEG): container finished" podID="0ee43833-c62b-4389-aab7-a95d43d43e12" containerID="ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325" exitCode=0 Nov 02 17:55:46.377500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.377467 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" event={"ID":"0ee43833-c62b-4389-aab7-a95d43d43e12","Type":"ContainerDied","Data":"ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325"} Nov 02 17:55:46.377722 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.377699 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" event={"ID":"0ee43833-c62b-4389-aab7-a95d43d43e12","Type":"ContainerDied","Data":"6473d8553c0d93438d3affaa0515c55f57f8c106e90e29ceb94c5ee2f9d094fb"} Nov 02 17:55:46.377875 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.377860 4310 scope.go:122] "RemoveContainer" containerID="ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325" Nov 02 17:55:46.378124 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.378109 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st" Nov 02 17:55:46.383746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.382925 4310 generic.go:356] "Generic (PLEG): container finished" podID="7e6468e5-cb25-4e25-9f18-7778166b5bed" containerID="7551e6d84507df41906e2a045f6eb959e74dbb413a2bccffdf3f88d49f1a58c1" exitCode=0 Nov 02 17:55:46.383746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.383055 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" event={"ID":"7e6468e5-cb25-4e25-9f18-7778166b5bed","Type":"ContainerDied","Data":"7551e6d84507df41906e2a045f6eb959e74dbb413a2bccffdf3f88d49f1a58c1"} Nov 02 17:55:46.383746 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.383272 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn" Nov 02 17:55:46.425454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.425423 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:46.426045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.426030 4310 scope.go:122] "RemoveContainer" containerID="ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325" Nov 02 17:55:46.426593 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:55:46.426572 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325\": container with ID starting with ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325 not found: ID does not exist" containerID="ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325" Nov 02 17:55:46.426760 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.426734 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325"} err="failed to get container status \"ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325\": rpc error: code = NotFound desc = could not find container \"ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325\": container with ID starting with ded310a61c7366da05e8a4152bf386f5374a1bdd024a98866a47df2d4969a325 not found: ID does not exist" Nov 02 17:55:46.426872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.426859 4310 scope.go:122] "RemoveContainer" containerID="7551e6d84507df41906e2a045f6eb959e74dbb413a2bccffdf3f88d49f1a58c1" Nov 02 17:55:46.524235 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.524206 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee43833-c62b-4389-aab7-a95d43d43e12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee43833-c62b-4389-aab7-a95d43d43e12-plugins-config-volume\") pod \"0ee43833-c62b-4389-aab7-a95d43d43e12\" (UID: \"0ee43833-c62b-4389-aab7-a95d43d43e12\") " Nov 02 17:55:46.524417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.524317 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee43833-c62b-4389-aab7-a95d43d43e12-kube-api-access-ct8ht\" (UniqueName: \"kubernetes.io/projected/0ee43833-c62b-4389-aab7-a95d43d43e12-kube-api-access-ct8ht\") pod \"0ee43833-c62b-4389-aab7-a95d43d43e12\" (UID: \"0ee43833-c62b-4389-aab7-a95d43d43e12\") " Nov 02 17:55:46.524417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.524394 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e6468e5-cb25-4e25-9f18-7778166b5bed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6468e5-cb25-4e25-9f18-7778166b5bed-plugins-config-volume\") pod \"7e6468e5-cb25-4e25-9f18-7778166b5bed\" (UID: \"7e6468e5-cb25-4e25-9f18-7778166b5bed\") " Nov 02 17:55:46.524518 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.524479 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6468e5-cb25-4e25-9f18-7778166b5bed-kube-api-access-hx5m7\" (UniqueName: \"kubernetes.io/projected/7e6468e5-cb25-4e25-9f18-7778166b5bed-kube-api-access-hx5m7\") pod \"7e6468e5-cb25-4e25-9f18-7778166b5bed\" (UID: \"7e6468e5-cb25-4e25-9f18-7778166b5bed\") " Nov 02 17:55:46.524600 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.524571 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee43833-c62b-4389-aab7-a95d43d43e12-plugins-config-volume" pod "0ee43833-c62b-4389-aab7-a95d43d43e12" (UID: "0ee43833-c62b-4389-aab7-a95d43d43e12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:46.524843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.524821 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6468e5-cb25-4e25-9f18-7778166b5bed-plugins-config-volume" pod "7e6468e5-cb25-4e25-9f18-7778166b5bed" (UID: "7e6468e5-cb25-4e25-9f18-7778166b5bed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:46.525337 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.525314 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee43833-c62b-4389-aab7-a95d43d43e12-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:46.525407 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.525340 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6468e5-cb25-4e25-9f18-7778166b5bed-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:46.526611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.526592 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6468e5-cb25-4e25-9f18-7778166b5bed-kube-api-access-hx5m7" pod "7e6468e5-cb25-4e25-9f18-7778166b5bed" (UID: "7e6468e5-cb25-4e25-9f18-7778166b5bed"). InnerVolumeSpecName "kube-api-access-hx5m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:46.526806 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.526778 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee43833-c62b-4389-aab7-a95d43d43e12-kube-api-access-ct8ht" pod "0ee43833-c62b-4389-aab7-a95d43d43e12" (UID: "0ee43833-c62b-4389-aab7-a95d43d43e12"). InnerVolumeSpecName "kube-api-access-ct8ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:46.626664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.626565 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct8ht\" (UniqueName: \"kubernetes.io/projected/0ee43833-c62b-4389-aab7-a95d43d43e12-kube-api-access-ct8ht\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:46.626664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.626604 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx5m7\" (UniqueName: \"kubernetes.io/projected/7e6468e5-cb25-4e25-9f18-7778166b5bed-kube-api-access-hx5m7\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:46.695055 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.695016 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" path="/var/lib/kubelet/pods/29d27280-8d99-4708-be3a-e4781a0c43b0/volumes" Nov 02 17:55:46.697539 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.697512 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599a96d5-51ba-49d6-aef7-b29e0a2dc27f" path="/var/lib/kubelet/pods/599a96d5-51ba-49d6-aef7-b29e0a2dc27f/volumes" Nov 02 17:55:46.736331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.736297 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st"] Nov 02 17:55:46.744906 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.744870 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d59fa283-69f86f5775-th2st"] Nov 02 17:55:46.758825 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.758794 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn"] Nov 02 17:55:46.767217 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.767189 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d59fa283-b85b6444c-drcnn"] Nov 02 17:55:46.868544 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.868511 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:46.991729 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:46.991694 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:47.105182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:47.105137 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:47.532768 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:47.525012 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:47.656953 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:47.656920 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:48.308408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:48.308371 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:48.677192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:48.677114 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee43833-c62b-4389-aab7-a95d43d43e12" path="/var/lib/kubelet/pods/0ee43833-c62b-4389-aab7-a95d43d43e12/volumes" Nov 02 17:55:48.677679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:48.677663 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e6468e5-cb25-4e25-9f18-7778166b5bed" path="/var/lib/kubelet/pods/7e6468e5-cb25-4e25-9f18-7778166b5bed/volumes" Nov 02 17:55:49.332096 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:49.325564 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:49.674225 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:49.673897 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:49.822098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:49.819318 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" podUID="8e897192-ee80-4702-8eb1-ea8b2f91cc55" containerName="payload-processing" containerID="cri-o://e73390ec203950e73e0a8039918b66969e8cd7ed85751a844243fbf88ba75e45" gracePeriod=25 Nov 02 17:55:49.865111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:49.822781 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" podUID="3f812cba-ade5-4cd3-bd87-fdea5395504b" containerName="payload-pre-processing" containerID="cri-o://5626d05de5524d2013eb11a7fae5a16499bdaf5c619ceb2cc77db8873edbc2c9" gracePeriod=25 Nov 02 17:55:49.951604 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:49.950169 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:50.039524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:50.039487 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b"] Nov 02 17:55:50.039769 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:50.039753 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51cd4868-mjh7b"] Nov 02 17:55:50.220637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:50.219842 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:50.341244 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:50.341155 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:50.899291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:50.896509 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b603fc8-c049-4a35-a1ad-9dbd487a2947" path="/var/lib/kubelet/pods/6b603fc8-c049-4a35-a1ad-9dbd487a2947/volumes" Nov 02 17:55:50.899291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:50.898571 4310 generic.go:356] "Generic (PLEG): container finished" podID="3f812cba-ade5-4cd3-bd87-fdea5395504b" containerID="5626d05de5524d2013eb11a7fae5a16499bdaf5c619ceb2cc77db8873edbc2c9" exitCode=0 Nov 02 17:55:50.899291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:50.898683 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" event={"ID":"3f812cba-ade5-4cd3-bd87-fdea5395504b","Type":"ContainerDied","Data":"5626d05de5524d2013eb11a7fae5a16499bdaf5c619ceb2cc77db8873edbc2c9"} Nov 02 17:55:50.906359 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:50.903783 4310 generic.go:356] "Generic (PLEG): container finished" podID="8e897192-ee80-4702-8eb1-ea8b2f91cc55" containerID="e73390ec203950e73e0a8039918b66969e8cd7ed85751a844243fbf88ba75e45" exitCode=0 Nov 02 17:55:50.906359 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:50.903886 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" event={"ID":"8e897192-ee80-4702-8eb1-ea8b2f91cc55","Type":"ContainerDied","Data":"e73390ec203950e73e0a8039918b66969e8cd7ed85751a844243fbf88ba75e45"} Nov 02 17:55:51.022553 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.022458 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:51.064284 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.062659 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:51.172104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.168069 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:51.224327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.223772 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" podUID="c467c116-1975-4e66-ab08-42ec962e170f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:55:51.230054 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.229568 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:51.230632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.230601 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" podUID="b66cbc84-3c7c-471b-914a-855635642fc8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:55:51.254855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.252793 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e897192-ee80-4702-8eb1-ea8b2f91cc55-kube-api-access-4j2wb\" (UniqueName: \"kubernetes.io/projected/8e897192-ee80-4702-8eb1-ea8b2f91cc55-kube-api-access-4j2wb\") pod \"8e897192-ee80-4702-8eb1-ea8b2f91cc55\" (UID: \"8e897192-ee80-4702-8eb1-ea8b2f91cc55\") " Nov 02 17:55:51.254855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.252957 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e897192-ee80-4702-8eb1-ea8b2f91cc55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e897192-ee80-4702-8eb1-ea8b2f91cc55-plugins-config-volume\") pod \"8e897192-ee80-4702-8eb1-ea8b2f91cc55\" (UID: \"8e897192-ee80-4702-8eb1-ea8b2f91cc55\") " Nov 02 17:55:51.256215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.255485 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e897192-ee80-4702-8eb1-ea8b2f91cc55-plugins-config-volume" pod "8e897192-ee80-4702-8eb1-ea8b2f91cc55" (UID: "8e897192-ee80-4702-8eb1-ea8b2f91cc55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:51.259666 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.259630 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e897192-ee80-4702-8eb1-ea8b2f91cc55-kube-api-access-4j2wb" pod "8e897192-ee80-4702-8eb1-ea8b2f91cc55" (UID: "8e897192-ee80-4702-8eb1-ea8b2f91cc55"). InnerVolumeSpecName "kube-api-access-4j2wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:51.260677 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.260651 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:55:51.349163 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.349032 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:51.354317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.354279 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f812cba-ade5-4cd3-bd87-fdea5395504b-kube-api-access-j4qgs\" (UniqueName: \"kubernetes.io/projected/3f812cba-ade5-4cd3-bd87-fdea5395504b-kube-api-access-j4qgs\") pod \"3f812cba-ade5-4cd3-bd87-fdea5395504b\" (UID: \"3f812cba-ade5-4cd3-bd87-fdea5395504b\") " Nov 02 17:55:51.354531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.354508 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-kube-api-access-5nc5r\" (UniqueName: \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-kube-api-access-5nc5r\") pod \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " Nov 02 17:55:51.354588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.354560 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-data\") pod \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " Nov 02 17:55:51.354655 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.354638 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-podinfo\") pod \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " Nov 02 17:55:51.354717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.354684 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-socket\") pod \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " Nov 02 17:55:51.354800 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.354783 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f812cba-ade5-4cd3-bd87-fdea5395504b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f812cba-ade5-4cd3-bd87-fdea5395504b-plugins-config-volume\") pod \"3f812cba-ade5-4cd3-bd87-fdea5395504b\" (UID: \"3f812cba-ade5-4cd3-bd87-fdea5395504b\") " Nov 02 17:55:51.354883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.354865 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-envoy\") pod \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " Nov 02 17:55:51.354953 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.354901 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-credential-socket\") pod \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " Nov 02 17:55:51.355001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.354954 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-token\" (UniqueName: \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-token\") pod \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " Nov 02 17:55:51.355055 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.355031 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istiod-ca-cert\") pod \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " Nov 02 17:55:51.355188 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.355170 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-certs\") pod \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\" (UID: \"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc\") " Nov 02 17:55:51.355939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.355598 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f812cba-ade5-4cd3-bd87-fdea5395504b-plugins-config-volume" pod "3f812cba-ade5-4cd3-bd87-fdea5395504b" (UID: "3f812cba-ade5-4cd3-bd87-fdea5395504b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:51.355939 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.355841 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-certs" pod "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" (UID: "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:51.356189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.356039 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-credential-socket" pod "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" (UID: "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:51.356865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.356835 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istiod-ca-cert" pod "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" (UID: "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:51.357119 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.357067 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j2wb\" (UniqueName: \"kubernetes.io/projected/8e897192-ee80-4702-8eb1-ea8b2f91cc55-kube-api-access-4j2wb\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.357232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.357128 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e897192-ee80-4702-8eb1-ea8b2f91cc55-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.357232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.357145 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f812cba-ade5-4cd3-bd87-fdea5395504b-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.357232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.357160 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.357232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.357174 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.357552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.357527 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-podinfo" pod "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" (UID: "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:51.357862 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.357839 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-data" pod "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" (UID: "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:51.358070 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.358048 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-socket" pod "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" (UID: "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:51.358587 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.358560 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f812cba-ade5-4cd3-bd87-fdea5395504b-kube-api-access-j4qgs" pod "3f812cba-ade5-4cd3-bd87-fdea5395504b" (UID: "3f812cba-ade5-4cd3-bd87-fdea5395504b"). InnerVolumeSpecName "kube-api-access-j4qgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:51.359265 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.359240 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-token" pod "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" (UID: "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:51.359354 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.359285 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-envoy" pod "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" (UID: "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:51.360396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.360376 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-kube-api-access-5nc5r" pod "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" (UID: "6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc"). InnerVolumeSpecName "kube-api-access-5nc5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:51.459417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.459312 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nc5r\" (UniqueName: \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-kube-api-access-5nc5r\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.459417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.459357 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.459417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.459374 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.459417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.459388 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.459417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.459403 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.459417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.459418 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.459866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.459431 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.459866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.459448 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4qgs\" (UniqueName: \"kubernetes.io/projected/3f812cba-ade5-4cd3-bd87-fdea5395504b-kube-api-access-j4qgs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:55:51.938147 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.938106 4310 generic.go:356] "Generic (PLEG): container finished" podID="6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" containerID="2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c" exitCode=0 Nov 02 17:55:51.939737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.939048 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" event={"ID":"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc","Type":"ContainerDied","Data":"2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c"} Nov 02 17:55:51.939737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.939133 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" event={"ID":"6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc","Type":"ContainerDied","Data":"355716f335b3a0ee6055d864881935d65e2c8f7fb707d85f1d55924a65cc9831"} Nov 02 17:55:51.939737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.939146 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk" Nov 02 17:55:51.939737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.939165 4310 scope.go:122] "RemoveContainer" containerID="2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c" Nov 02 17:55:51.941995 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.941967 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" event={"ID":"3f812cba-ade5-4cd3-bd87-fdea5395504b","Type":"ContainerDied","Data":"3ad0f24f9f55b3d17b0d1f43d329dbd641f451254264a415bc1a649425f15d7f"} Nov 02 17:55:51.964136 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.942131 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg" Nov 02 17:55:51.964136 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.963233 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" event={"ID":"8e897192-ee80-4702-8eb1-ea8b2f91cc55","Type":"ContainerDied","Data":"209ce6f8926a341a79ea2c7cd6c5ed12919a9cd61e62e7e379992fb5ced211e5"} Nov 02 17:55:51.964136 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:51.963343 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w" Nov 02 17:55:52.032247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.032046 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:52.286159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.241763 4310 scope.go:122] "RemoveContainer" containerID="2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c" Nov 02 17:55:52.286159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.241925 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg"] Nov 02 17:55:52.287849 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:55:52.287557 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c\": container with ID starting with 2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c not found: ID does not exist" containerID="2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c" Nov 02 17:55:52.287849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.287620 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c"} err="failed to get container status \"2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c\": rpc error: code = NotFound desc = could not find container \"2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c\": container with ID starting with 2e576b604a4c64e744d21341c89fce51f1b15498a562e8711b9cf235c0e1350c not found: ID does not exist" Nov 02 17:55:52.287849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.287660 4310 scope.go:122] "RemoveContainer" containerID="5626d05de5524d2013eb11a7fae5a16499bdaf5c619ceb2cc77db8873edbc2c9" Nov 02 17:55:52.374311 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.374220 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51cd4868-69886fdb7f-c7rmg"] Nov 02 17:55:52.432531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.432497 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w"] Nov 02 17:55:52.437069 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.437048 4310 scope.go:122] "RemoveContainer" containerID="e73390ec203950e73e0a8039918b66969e8cd7ed85751a844243fbf88ba75e45" Nov 02 17:55:52.529769 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.529602 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51cd4868-848ddbbb77-glm6w"] Nov 02 17:55:52.630493 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.630298 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk"] Nov 02 17:55:52.642596 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.642560 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d59fa283-openshift-default-758ff6bb9c-qsfkk"] Nov 02 17:55:52.739407 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.738643 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f812cba-ade5-4cd3-bd87-fdea5395504b" path="/var/lib/kubelet/pods/3f812cba-ade5-4cd3-bd87-fdea5395504b/volumes" Nov 02 17:55:52.739407 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.739326 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" path="/var/lib/kubelet/pods/6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc/volumes" Nov 02 17:55:52.739855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:52.739837 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e897192-ee80-4702-8eb1-ea8b2f91cc55" path="/var/lib/kubelet/pods/8e897192-ee80-4702-8eb1-ea8b2f91cc55/volumes" Nov 02 17:55:53.140124 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:53.139191 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:53.758102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:53.749830 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:53.901171 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:53.901046 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:53.965195 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:53.965155 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2"] Nov 02 17:55:53.994932 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:53.994895 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7b0683-kvmb2"] Nov 02 17:55:54.170711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:54.170622 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:54.424566 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:54.424484 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:54.678550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:54.678469 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="812f2a44-c138-4ca1-89cf-f0b4dd596322" path="/var/lib/kubelet/pods/812f2a44-c138-4ca1-89cf-f0b4dd596322/volumes" Nov 02 17:55:55.042446 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:55.042376 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:55.145177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:55.144630 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:55.794882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:55.794843 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:56.584512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:56.584481 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:56.687397 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:56.685174 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:57.659005 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:57.658966 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:57.809034 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:57.809004 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:58.629908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:58.629864 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:59.620691 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.620653 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:59.623967 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.623938 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6"] Nov 02 17:55:59.625140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625120 4310 cpu_manager.go:414] "containerMap: removing container" podUID="8e897192-ee80-4702-8eb1-ea8b2f91cc55" containerName="payload-processing" Nov 02 17:55:59.625197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625144 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e897192-ee80-4702-8eb1-ea8b2f91cc55" containerName="payload-processing" Nov 02 17:55:59.625197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625176 4310 cpu_manager.go:414] "containerMap: removing container" podUID="6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" containerName="istio-proxy" Nov 02 17:55:59.625197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625185 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" containerName="istio-proxy" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625207 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a890090c-9e21-4253-a027-05438d80f288" containerName="payload-pre-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625216 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a890090c-9e21-4253-a027-05438d80f288" containerName="payload-pre-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625234 4310 cpu_manager.go:414] "containerMap: removing container" podUID="3f812cba-ade5-4cd3-bd87-fdea5395504b" containerName="payload-pre-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625242 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f812cba-ade5-4cd3-bd87-fdea5395504b" containerName="payload-pre-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625264 4310 cpu_manager.go:414] "containerMap: removing container" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625271 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625295 4310 cpu_manager.go:414] "containerMap: removing container" podUID="7e6468e5-cb25-4e25-9f18-7778166b5bed" containerName="payload-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625302 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6468e5-cb25-4e25-9f18-7778166b5bed" containerName="payload-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625315 4310 cpu_manager.go:414] "containerMap: removing container" podUID="1103c0e1-74d5-4035-ba22-f56bd12f16dd" containerName="payload-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625323 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1103c0e1-74d5-4035-ba22-f56bd12f16dd" containerName="payload-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625334 4310 cpu_manager.go:414] "containerMap: removing container" podUID="6b603fc8-c049-4a35-a1ad-9dbd487a2947" containerName="revoke-keys" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625367 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b603fc8-c049-4a35-a1ad-9dbd487a2947" containerName="revoke-keys" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625379 4310 cpu_manager.go:414] "containerMap: removing container" podUID="c3ef6ff4-a23e-45b0-900e-3c3c057df7cd" containerName="revoke-keys" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625388 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ef6ff4-a23e-45b0-900e-3c3c057df7cd" containerName="revoke-keys" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625428 4310 cpu_manager.go:414] "containerMap: removing container" podUID="0ee43833-c62b-4389-aab7-a95d43d43e12" containerName="payload-pre-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625435 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee43833-c62b-4389-aab7-a95d43d43e12" containerName="payload-pre-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625456 4310 cpu_manager.go:414] "containerMap: removing container" podUID="69353ed1-5667-4407-a4bc-36105ebcb410" containerName="revoke-keys" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625463 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69353ed1-5667-4407-a4bc-36105ebcb410" containerName="revoke-keys" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625653 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e897192-ee80-4702-8eb1-ea8b2f91cc55" containerName="payload-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625671 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dfc8f71-82b9-4c5e-b7e4-b7d6cce90ddc" containerName="istio-proxy" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625683 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a890090c-9e21-4253-a027-05438d80f288" containerName="payload-pre-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625697 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f812cba-ade5-4cd3-bd87-fdea5395504b" containerName="payload-pre-processing" Nov 02 17:55:59.625701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625711 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d27280-8d99-4708-be3a-e4781a0c43b0" containerName="main" Nov 02 17:55:59.626571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625722 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6468e5-cb25-4e25-9f18-7778166b5bed" containerName="payload-processing" Nov 02 17:55:59.626571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625732 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="1103c0e1-74d5-4035-ba22-f56bd12f16dd" containerName="payload-processing" Nov 02 17:55:59.626571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625740 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b603fc8-c049-4a35-a1ad-9dbd487a2947" containerName="revoke-keys" Nov 02 17:55:59.626571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625750 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ef6ff4-a23e-45b0-900e-3c3c057df7cd" containerName="revoke-keys" Nov 02 17:55:59.626571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625763 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee43833-c62b-4389-aab7-a95d43d43e12" containerName="payload-pre-processing" Nov 02 17:55:59.626571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.625773 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="69353ed1-5667-4407-a4bc-36105ebcb410" containerName="revoke-keys" Nov 02 17:55:59.637550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.636990 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:55:59.642352 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.642321 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6"] Nov 02 17:55:59.781444 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.695571 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:55:59.781444 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.695700 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:55:59.781444 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.781335 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltgwf\" (UniqueName: \"kubernetes.io/projected/acd6a9c0-5175-4140-9682-6d369c7b7825-kube-api-access-ltgwf\") pod \"maas-api-revoke-sub-140daf35-nhsh6\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:55:59.781852 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.781456 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140daf35-nhsh6\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:55:59.781852 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.781533 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140daf35-nhsh6\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:55:59.807476 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.807351 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:55:59.910802 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.910102 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgwf\" (UniqueName: \"kubernetes.io/projected/acd6a9c0-5175-4140-9682-6d369c7b7825-kube-api-access-ltgwf\") pod \"maas-api-revoke-sub-140daf35-nhsh6\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:55:59.910802 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.910248 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140daf35-nhsh6\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:55:59.910802 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.910326 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140daf35-nhsh6\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:55:59.912353 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.912310 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140daf35-nhsh6\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:55:59.917813 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.917787 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140daf35-nhsh6\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:55:59.919884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.919845 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgwf\" (UniqueName: \"kubernetes.io/projected/acd6a9c0-5175-4140-9682-6d369c7b7825-kube-api-access-ltgwf\") pod \"maas-api-revoke-sub-140daf35-nhsh6\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:55:59.974936 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:55:59.974183 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:00.079969 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.079526 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:56:00.127614 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.127571 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:56:00.213065 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.213006 4310 generic.go:356] "Generic (PLEG): container finished" podID="b66cbc84-3c7c-471b-914a-855635642fc8" containerID="3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af" exitCode=0 Nov 02 17:56:00.213254 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.213158 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" event={"ID":"b66cbc84-3c7c-471b-914a-855635642fc8","Type":"ContainerDied","Data":"3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af"} Nov 02 17:56:00.213254 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.213230 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" event={"ID":"b66cbc84-3c7c-471b-914a-855635642fc8","Type":"ContainerDied","Data":"d5f729369ee47fc79af7a8799cec49ce38b5028a3cd31c2818ccef1ebbbecf92"} Nov 02 17:56:00.213407 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.213265 4310 scope.go:122] "RemoveContainer" containerID="3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af" Nov 02 17:56:00.213700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.213676 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9" Nov 02 17:56:00.214063 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.214045 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:56:00.215423 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.215400 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66cbc84-3c7c-471b-914a-855635642fc8-kube-api-access-24nsc\" (UniqueName: \"kubernetes.io/projected/b66cbc84-3c7c-471b-914a-855635642fc8-kube-api-access-24nsc\") pod \"b66cbc84-3c7c-471b-914a-855635642fc8\" (UID: \"b66cbc84-3c7c-471b-914a-855635642fc8\") " Nov 02 17:56:00.215542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.215525 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66cbc84-3c7c-471b-914a-855635642fc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66cbc84-3c7c-471b-914a-855635642fc8-plugins-config-volume\") pod \"b66cbc84-3c7c-471b-914a-855635642fc8\" (UID: \"b66cbc84-3c7c-471b-914a-855635642fc8\") " Nov 02 17:56:00.215777 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.215759 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c467c116-1975-4e66-ab08-42ec962e170f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c467c116-1975-4e66-ab08-42ec962e170f-plugins-config-volume\") pod \"c467c116-1975-4e66-ab08-42ec962e170f\" (UID: \"c467c116-1975-4e66-ab08-42ec962e170f\") " Nov 02 17:56:00.215891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.215868 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c467c116-1975-4e66-ab08-42ec962e170f-kube-api-access-rb727\" (UniqueName: \"kubernetes.io/projected/c467c116-1975-4e66-ab08-42ec962e170f-kube-api-access-rb727\") pod \"c467c116-1975-4e66-ab08-42ec962e170f\" (UID: \"c467c116-1975-4e66-ab08-42ec962e170f\") " Nov 02 17:56:00.218306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.217381 4310 generic.go:356] "Generic (PLEG): container finished" podID="c467c116-1975-4e66-ab08-42ec962e170f" containerID="5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003" exitCode=0 Nov 02 17:56:00.218306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.217729 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" Nov 02 17:56:00.218306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.218039 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" event={"ID":"c467c116-1975-4e66-ab08-42ec962e170f","Type":"ContainerDied","Data":"5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003"} Nov 02 17:56:00.218306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.218150 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh" event={"ID":"c467c116-1975-4e66-ab08-42ec962e170f","Type":"ContainerDied","Data":"251cf644a65cf5b78e18eb05c1af05d9ea7b13166609c85a0d7cb101424605a4"} Nov 02 17:56:00.221144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.219176 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c467c116-1975-4e66-ab08-42ec962e170f-kube-api-access-rb727" pod "c467c116-1975-4e66-ab08-42ec962e170f" (UID: "c467c116-1975-4e66-ab08-42ec962e170f"). InnerVolumeSpecName "kube-api-access-rb727". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:00.221144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.219490 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66cbc84-3c7c-471b-914a-855635642fc8-plugins-config-volume" pod "b66cbc84-3c7c-471b-914a-855635642fc8" (UID: "b66cbc84-3c7c-471b-914a-855635642fc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:00.221144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.219502 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c467c116-1975-4e66-ab08-42ec962e170f-plugins-config-volume" pod "c467c116-1975-4e66-ab08-42ec962e170f" (UID: "c467c116-1975-4e66-ab08-42ec962e170f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:00.221144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.220260 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66cbc84-3c7c-471b-914a-855635642fc8-kube-api-access-24nsc" pod "b66cbc84-3c7c-471b-914a-855635642fc8" (UID: "b66cbc84-3c7c-471b-914a-855635642fc8"). InnerVolumeSpecName "kube-api-access-24nsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:00.234845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.234803 4310 scope.go:122] "RemoveContainer" containerID="3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af" Nov 02 17:56:00.235333 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:56:00.235299 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af\": container with ID starting with 3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af not found: ID does not exist" containerID="3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af" Nov 02 17:56:00.235425 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.235345 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af"} err="failed to get container status \"3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af\": rpc error: code = NotFound desc = could not find container \"3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af\": container with ID starting with 3c7d0e071360aed9a6872b8e02765223a7280f093496c7c0f216b7bbf12ce7af not found: ID does not exist" Nov 02 17:56:00.235425 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.235371 4310 scope.go:122] "RemoveContainer" containerID="5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003" Nov 02 17:56:00.254009 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.253888 4310 scope.go:122] "RemoveContainer" containerID="5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003" Nov 02 17:56:00.254617 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:56:00.254592 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003\": container with ID starting with 5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003 not found: ID does not exist" containerID="5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003" Nov 02 17:56:00.254729 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.254631 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003"} err="failed to get container status \"5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003\": rpc error: code = NotFound desc = could not find container \"5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003\": container with ID starting with 5dbdd594d39759ce27fe0598ac13913824aae9f85b004d16af2bf8838dd68003 not found: ID does not exist" Nov 02 17:56:00.348681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.348570 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c467c116-1975-4e66-ab08-42ec962e170f-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:00.348681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.348612 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb727\" (UniqueName: \"kubernetes.io/projected/c467c116-1975-4e66-ab08-42ec962e170f-kube-api-access-rb727\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:00.348681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.348626 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24nsc\" (UniqueName: \"kubernetes.io/projected/b66cbc84-3c7c-471b-914a-855635642fc8-kube-api-access-24nsc\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:00.348681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.348639 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66cbc84-3c7c-471b-914a-855635642fc8-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:00.668480 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:00.638439 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:01.152332 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.152294 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:01.178304 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.176357 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42"] Nov 02 17:56:01.178304 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.177953 4310 cpu_manager.go:414] "containerMap: removing container" podUID="b66cbc84-3c7c-471b-914a-855635642fc8" containerName="payload-processing" Nov 02 17:56:01.178304 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.177975 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66cbc84-3c7c-471b-914a-855635642fc8" containerName="payload-processing" Nov 02 17:56:01.178304 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.178105 4310 cpu_manager.go:414] "containerMap: removing container" podUID="c467c116-1975-4e66-ab08-42ec962e170f" containerName="payload-pre-processing" Nov 02 17:56:01.178304 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.178116 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c467c116-1975-4e66-ab08-42ec962e170f" containerName="payload-pre-processing" Nov 02 17:56:01.178681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.178465 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="c467c116-1975-4e66-ab08-42ec962e170f" containerName="payload-pre-processing" Nov 02 17:56:01.178681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.178485 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66cbc84-3c7c-471b-914a-855635642fc8" containerName="payload-processing" Nov 02 17:56:01.186399 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.186372 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42"] Nov 02 17:56:01.186643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.186612 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:01.196986 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.196956 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-33a0a14d\"" Nov 02 17:56:01.197169 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.197053 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-33a0a14d-dockercfg-bhrsd\"" Nov 02 17:56:01.200965 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.200943 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh"] Nov 02 17:56:01.210457 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.210438 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7b0683-f6545f9ff-sdbzh"] Nov 02 17:56:01.219779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.219754 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9"] Nov 02 17:56:01.229872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.229844 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7b0683-fd76fbd8c-h26w9"] Nov 02 17:56:01.242424 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:56:01.242397 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd6a9c0_5175_4140_9682_6d369c7b7825.slice/crio-16f037489caed044994631d271576cd431d903c396d73a35452cf98a0449ca9c WatchSource:0}: Error finding container 16f037489caed044994631d271576cd431d903c396d73a35452cf98a0449ca9c: Status 404 returned error can't find the container with id 16f037489caed044994631d271576cd431d903c396d73a35452cf98a0449ca9c Nov 02 17:56:01.244802 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.244780 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6"] Nov 02 17:56:01.358931 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.358890 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8hb\" (UniqueName: \"kubernetes.io/projected/c97b6368-9d5a-4eea-bc51-38db653b0396-kube-api-access-fx8hb\") pod \"payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42\" (UID: \"c97b6368-9d5a-4eea-bc51-38db653b0396\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:01.359496 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.359466 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97b6368-9d5a-4eea-bc51-38db653b0396-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42\" (UID: \"c97b6368-9d5a-4eea-bc51-38db653b0396\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:01.460616 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.460588 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97b6368-9d5a-4eea-bc51-38db653b0396-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42\" (UID: \"c97b6368-9d5a-4eea-bc51-38db653b0396\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:01.460749 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.460733 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8hb\" (UniqueName: \"kubernetes.io/projected/c97b6368-9d5a-4eea-bc51-38db653b0396-kube-api-access-fx8hb\") pod \"payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42\" (UID: \"c97b6368-9d5a-4eea-bc51-38db653b0396\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:01.462176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.461857 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97b6368-9d5a-4eea-bc51-38db653b0396-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42\" (UID: \"c97b6368-9d5a-4eea-bc51-38db653b0396\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:01.472723 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.472688 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8hb\" (UniqueName: \"kubernetes.io/projected/c97b6368-9d5a-4eea-bc51-38db653b0396-kube-api-access-fx8hb\") pod \"payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42\" (UID: \"c97b6368-9d5a-4eea-bc51-38db653b0396\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:01.518004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.517969 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:01.544501 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.544449 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:01.820093 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.820038 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:01.957700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.954059 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:01.966131 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:01.966100 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42"] Nov 02 17:56:02.200656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.200161 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b"] Nov 02 17:56:02.200656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.200383 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" containerID="cri-o://de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957" gracePeriod=30 Nov 02 17:56:02.207828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.207293 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn"] Nov 02 17:56:02.207828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.207519 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" containerID="cri-o://64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc" gracePeriod=30 Nov 02 17:56:02.214771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.214749 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc"] Nov 02 17:56:02.214917 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.214901 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" containerID="cri-o://0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5" gracePeriod=30 Nov 02 17:56:02.222381 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.222358 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:02.247771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.247744 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" event={"ID":"acd6a9c0-5175-4140-9682-6d369c7b7825","Type":"ContainerStarted","Data":"fb07e144101c63f16153c1311f099fa8e245830ff75606fb13892c88f0ff3695"} Nov 02 17:56:02.249271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.248466 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" event={"ID":"acd6a9c0-5175-4140-9682-6d369c7b7825","Type":"ContainerStarted","Data":"16f037489caed044994631d271576cd431d903c396d73a35452cf98a0449ca9c"} Nov 02 17:56:02.251347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.251320 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" event={"ID":"c97b6368-9d5a-4eea-bc51-38db653b0396","Type":"ContainerStarted","Data":"42268310903d7fb1417c00a4310cc85203f1058192fe0096d85456ebce116c76"} Nov 02 17:56:02.251439 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.251426 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:02.251490 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.251453 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" event={"ID":"c97b6368-9d5a-4eea-bc51-38db653b0396","Type":"ContainerStarted","Data":"f89fdcc99a2655a8e6e0918dfe4de636a13c0731979c4b558ba2944d1069d5c8"} Nov 02 17:56:02.269881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.269853 4310 patch_prober.go:29] interesting pod/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:56:02.270005 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.269909 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:56:02.322101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.317245 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" podStartSLOduration=2.317224606 podStartE2EDuration="2.317224606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:02.315921786 +0000 UTC m=+959.759004437" watchObservedRunningTime="2026-11-02 17:56:02.317224606 +0000 UTC m=+959.760307242" Nov 02 17:56:02.478181 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.461841 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:02.478181 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.470010 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" podStartSLOduration=3.469987198 podStartE2EDuration="3.469987198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:02.348679839 +0000 UTC m=+959.791762486" watchObservedRunningTime="2026-11-02 17:56:02.469987198 +0000 UTC m=+959.913069835" Nov 02 17:56:02.809178 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:02.790276 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:03.056879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:03.056841 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66cbc84-3c7c-471b-914a-855635642fc8" path="/var/lib/kubelet/pods/b66cbc84-3c7c-471b-914a-855635642fc8/volumes" Nov 02 17:56:03.058397 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:03.058371 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c467c116-1975-4e66-ab08-42ec962e170f" path="/var/lib/kubelet/pods/c467c116-1975-4e66-ab08-42ec962e170f/volumes" Nov 02 17:56:03.265438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:03.265398 4310 generic.go:356] "Generic (PLEG): container finished" podID="acd6a9c0-5175-4140-9682-6d369c7b7825" containerID="fb07e144101c63f16153c1311f099fa8e245830ff75606fb13892c88f0ff3695" exitCode=0 Nov 02 17:56:03.266173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:03.266138 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" event={"ID":"acd6a9c0-5175-4140-9682-6d369c7b7825","Type":"ContainerDied","Data":"fb07e144101c63f16153c1311f099fa8e245830ff75606fb13892c88f0ff3695"} Nov 02 17:56:03.488373 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:03.488324 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:03.569101 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:56:03.560146 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:56:04.277632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:04.277596 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:04.561837 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:04.561811 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:56:04.608943 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:04.608911 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-xks-ca\") pod \"acd6a9c0-5175-4140-9682-6d369c7b7825\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " Nov 02 17:56:04.609227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:04.609209 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd6a9c0-5175-4140-9682-6d369c7b7825-kube-api-access-ltgwf\" (UniqueName: \"kubernetes.io/projected/acd6a9c0-5175-4140-9682-6d369c7b7825-kube-api-access-ltgwf\") pod \"acd6a9c0-5175-4140-9682-6d369c7b7825\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " Nov 02 17:56:04.609377 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:04.609330 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-openshift-ca\") pod \"acd6a9c0-5175-4140-9682-6d369c7b7825\" (UID: \"acd6a9c0-5175-4140-9682-6d369c7b7825\") " Nov 02 17:56:04.609879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:04.609848 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-openshift-ca" pod "acd6a9c0-5175-4140-9682-6d369c7b7825" (UID: "acd6a9c0-5175-4140-9682-6d369c7b7825"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:04.610233 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:04.610207 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:04.611542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:04.611519 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-xks-ca" pod "acd6a9c0-5175-4140-9682-6d369c7b7825" (UID: "acd6a9c0-5175-4140-9682-6d369c7b7825"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:04.611542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:04.611534 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd6a9c0-5175-4140-9682-6d369c7b7825-kube-api-access-ltgwf" pod "acd6a9c0-5175-4140-9682-6d369c7b7825" (UID: "acd6a9c0-5175-4140-9682-6d369c7b7825"). InnerVolumeSpecName "kube-api-access-ltgwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:04.712153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:04.712115 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltgwf\" (UniqueName: \"kubernetes.io/projected/acd6a9c0-5175-4140-9682-6d369c7b7825-kube-api-access-ltgwf\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:04.712153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:04.712143 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd6a9c0-5175-4140-9682-6d369c7b7825-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:05.017679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:05.017644 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:05.288034 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:05.287955 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" Nov 02 17:56:05.288034 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:05.287986 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6" event={"ID":"acd6a9c0-5175-4140-9682-6d369c7b7825","Type":"ContainerDied","Data":"16f037489caed044994631d271576cd431d903c396d73a35452cf98a0449ca9c"} Nov 02 17:56:05.288445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:05.288038 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16f037489caed044994631d271576cd431d903c396d73a35452cf98a0449ca9c" Nov 02 17:56:05.614714 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:05.614626 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:05.740639 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:05.740600 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:06.408190 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:06.408141 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:07.194295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.194254 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:07.521266 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.521202 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:07.595916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.595885 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:56:07.636716 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.636688 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:56:07.676358 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.675612 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:56:07.727354 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727323 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-kube-api-access-x22mm\" (UniqueName: \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-kube-api-access-x22mm\") pod \"74414d9b-bc66-443d-89b6-03901d5b7890\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " Nov 02 17:56:07.727548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727413 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1801c5bf-b709-48b0-9e6b-721f85eb4520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1801c5bf-b709-48b0-9e6b-721f85eb4520-istiod-ca-cert\") pod \"1801c5bf-b709-48b0-9e6b-721f85eb4520\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " Nov 02 17:56:07.727548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727486 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-token\" (UniqueName: \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-token\") pod \"1801c5bf-b709-48b0-9e6b-721f85eb4520\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " Nov 02 17:56:07.727548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727521 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-envoy\") pod \"74414d9b-bc66-443d-89b6-03901d5b7890\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " Nov 02 17:56:07.727740 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727562 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istiod-ca-cert\") pod \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " Nov 02 17:56:07.727740 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727686 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-podinfo\") pod \"1801c5bf-b709-48b0-9e6b-721f85eb4520\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " Nov 02 17:56:07.727740 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727729 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-data\") pod \"1801c5bf-b709-48b0-9e6b-721f85eb4520\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " Nov 02 17:56:07.727907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727792 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-kube-api-access-mzq8f\" (UniqueName: \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-kube-api-access-mzq8f\") pod \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " Nov 02 17:56:07.727907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727821 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-istio-token\" (UniqueName: \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-istio-token\") pod \"74414d9b-bc66-443d-89b6-03901d5b7890\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " Nov 02 17:56:07.727907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727830 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1801c5bf-b709-48b0-9e6b-721f85eb4520-istiod-ca-cert" pod "1801c5bf-b709-48b0-9e6b-721f85eb4520" (UID: "1801c5bf-b709-48b0-9e6b-721f85eb4520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:07.727907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727850 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-envoy\") pod \"1801c5bf-b709-48b0-9e6b-721f85eb4520\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " Nov 02 17:56:07.727907 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727891 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-socket\") pod \"74414d9b-bc66-443d-89b6-03901d5b7890\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " Nov 02 17:56:07.728174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727940 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-credential-socket\") pod \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " Nov 02 17:56:07.728174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.727972 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-certs\") pod \"74414d9b-bc66-443d-89b6-03901d5b7890\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " Nov 02 17:56:07.728174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728021 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-token\" (UniqueName: \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-token\") pod \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " Nov 02 17:56:07.728174 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728161 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74414d9b-bc66-443d-89b6-03901d5b7890-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74414d9b-bc66-443d-89b6-03901d5b7890-istio-podinfo\") pod \"74414d9b-bc66-443d-89b6-03901d5b7890\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " Nov 02 17:56:07.728388 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728193 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-data" pod "1801c5bf-b709-48b0-9e6b-721f85eb4520" (UID: "1801c5bf-b709-48b0-9e6b-721f85eb4520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.728388 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728242 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-certs\") pod \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " Nov 02 17:56:07.728388 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728294 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-envoy\") pod \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " Nov 02 17:56:07.728388 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728340 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-kube-api-access-wfkmj\" (UniqueName: \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-kube-api-access-wfkmj\") pod \"1801c5bf-b709-48b0-9e6b-721f85eb4520\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " Nov 02 17:56:07.728388 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728372 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-credential-socket\") pod \"74414d9b-bc66-443d-89b6-03901d5b7890\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " Nov 02 17:56:07.728681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728444 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-podinfo\") pod \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " Nov 02 17:56:07.728681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728475 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-socket\") pod \"1801c5bf-b709-48b0-9e6b-721f85eb4520\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " Nov 02 17:56:07.728681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728510 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74414d9b-bc66-443d-89b6-03901d5b7890-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74414d9b-bc66-443d-89b6-03901d5b7890-istiod-ca-cert\") pod \"74414d9b-bc66-443d-89b6-03901d5b7890\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " Nov 02 17:56:07.728681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728513 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istiod-ca-cert" pod "be6a1e76-95ae-4ef4-9123-b2518eee56a0" (UID: "be6a1e76-95ae-4ef4-9123-b2518eee56a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:07.728681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728543 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-certs\") pod \"1801c5bf-b709-48b0-9e6b-721f85eb4520\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " Nov 02 17:56:07.728681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728583 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-data\") pod \"74414d9b-bc66-443d-89b6-03901d5b7890\" (UID: \"74414d9b-bc66-443d-89b6-03901d5b7890\") " Nov 02 17:56:07.728681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728613 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-data\") pod \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " Nov 02 17:56:07.728681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728641 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-socket\") pod \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\" (UID: \"be6a1e76-95ae-4ef4-9123-b2518eee56a0\") " Nov 02 17:56:07.728681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.728669 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-credential-socket\") pod \"1801c5bf-b709-48b0-9e6b-721f85eb4520\" (UID: \"1801c5bf-b709-48b0-9e6b-721f85eb4520\") " Nov 02 17:56:07.729608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.729567 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1801c5bf-b709-48b0-9e6b-721f85eb4520-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.729608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.729597 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.730021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.729613 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.730021 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.729838 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-credential-socket" pod "1801c5bf-b709-48b0-9e6b-721f85eb4520" (UID: "1801c5bf-b709-48b0-9e6b-721f85eb4520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.730232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.730041 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-socket" pod "74414d9b-bc66-443d-89b6-03901d5b7890" (UID: "74414d9b-bc66-443d-89b6-03901d5b7890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.730341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.730284 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-credential-socket" pod "be6a1e76-95ae-4ef4-9123-b2518eee56a0" (UID: "be6a1e76-95ae-4ef4-9123-b2518eee56a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.730476 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.730455 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-certs" pod "74414d9b-bc66-443d-89b6-03901d5b7890" (UID: "74414d9b-bc66-443d-89b6-03901d5b7890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.732194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.732002 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-certs" pod "be6a1e76-95ae-4ef4-9123-b2518eee56a0" (UID: "be6a1e76-95ae-4ef4-9123-b2518eee56a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.732417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.732346 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-credential-socket" pod "74414d9b-bc66-443d-89b6-03901d5b7890" (UID: "74414d9b-bc66-443d-89b6-03901d5b7890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.732507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.732455 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-certs" pod "1801c5bf-b709-48b0-9e6b-721f85eb4520" (UID: "1801c5bf-b709-48b0-9e6b-721f85eb4520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.732913 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.732737 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-socket" pod "1801c5bf-b709-48b0-9e6b-721f85eb4520" (UID: "1801c5bf-b709-48b0-9e6b-721f85eb4520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.732913 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.732806 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74414d9b-bc66-443d-89b6-03901d5b7890-istiod-ca-cert" pod "74414d9b-bc66-443d-89b6-03901d5b7890" (UID: "74414d9b-bc66-443d-89b6-03901d5b7890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:07.733070 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.733036 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-data" pod "be6a1e76-95ae-4ef4-9123-b2518eee56a0" (UID: "be6a1e76-95ae-4ef4-9123-b2518eee56a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.733258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.733216 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-socket" pod "be6a1e76-95ae-4ef4-9123-b2518eee56a0" (UID: "be6a1e76-95ae-4ef4-9123-b2518eee56a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.733258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.733209 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-data" pod "74414d9b-bc66-443d-89b6-03901d5b7890" (UID: "74414d9b-bc66-443d-89b6-03901d5b7890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.734235 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.734207 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-kube-api-access-wfkmj" pod "1801c5bf-b709-48b0-9e6b-721f85eb4520" (UID: "1801c5bf-b709-48b0-9e6b-721f85eb4520"). InnerVolumeSpecName "kube-api-access-wfkmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.735728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.735701 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-envoy" pod "be6a1e76-95ae-4ef4-9123-b2518eee56a0" (UID: "be6a1e76-95ae-4ef4-9123-b2518eee56a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.735839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.735731 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-podinfo" pod "1801c5bf-b709-48b0-9e6b-721f85eb4520" (UID: "1801c5bf-b709-48b0-9e6b-721f85eb4520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:07.736194 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.736160 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-kube-api-access-mzq8f" pod "be6a1e76-95ae-4ef4-9123-b2518eee56a0" (UID: "be6a1e76-95ae-4ef4-9123-b2518eee56a0"). InnerVolumeSpecName "kube-api-access-mzq8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.736607 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.736575 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-podinfo" pod "be6a1e76-95ae-4ef4-9123-b2518eee56a0" (UID: "be6a1e76-95ae-4ef4-9123-b2518eee56a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:07.736798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.736779 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-token" pod "be6a1e76-95ae-4ef4-9123-b2518eee56a0" (UID: "be6a1e76-95ae-4ef4-9123-b2518eee56a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.737247 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.737216 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74414d9b-bc66-443d-89b6-03901d5b7890-istio-podinfo" pod "74414d9b-bc66-443d-89b6-03901d5b7890" (UID: "74414d9b-bc66-443d-89b6-03901d5b7890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:07.738283 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.737444 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-kube-api-access-x22mm" pod "74414d9b-bc66-443d-89b6-03901d5b7890" (UID: "74414d9b-bc66-443d-89b6-03901d5b7890"). InnerVolumeSpecName "kube-api-access-x22mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.738283 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.737468 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-envoy" pod "1801c5bf-b709-48b0-9e6b-721f85eb4520" (UID: "1801c5bf-b709-48b0-9e6b-721f85eb4520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.738283 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.737513 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-istio-token" pod "74414d9b-bc66-443d-89b6-03901d5b7890" (UID: "74414d9b-bc66-443d-89b6-03901d5b7890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.738283 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.737885 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-token" pod "1801c5bf-b709-48b0-9e6b-721f85eb4520" (UID: "1801c5bf-b709-48b0-9e6b-721f85eb4520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:07.739269 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.739243 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-envoy" pod "74414d9b-bc66-443d-89b6-03901d5b7890" (UID: "74414d9b-bc66-443d-89b6-03901d5b7890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:07.831883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831851 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.831883 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831886 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfkmj\" (UniqueName: \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-kube-api-access-wfkmj\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831902 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831915 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831928 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831937 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74414d9b-bc66-443d-89b6-03901d5b7890-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831946 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831956 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831965 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831977 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831989 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.831999 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x22mm\" (UniqueName: \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-kube-api-access-x22mm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832007 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832020 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832030 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832038 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzq8f\" (UniqueName: \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-kube-api-access-mzq8f\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832046 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74414d9b-bc66-443d-89b6-03901d5b7890-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832057 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1801c5bf-b709-48b0-9e6b-721f85eb4520-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832069 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832094 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832107 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74414d9b-bc66-443d-89b6-03901d5b7890-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832772 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832121 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6a1e76-95ae-4ef4-9123-b2518eee56a0-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832772 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832134 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74414d9b-bc66-443d-89b6-03901d5b7890-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.832772 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.832147 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6a1e76-95ae-4ef4-9123-b2518eee56a0-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:07.868583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:07.868555 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:08.106294 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.106188 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:08.379628 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.379549 4310 generic.go:356] "Generic (PLEG): container finished" podID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerID="de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957" exitCode=0 Nov 02 17:56:08.379628 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.379599 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" Nov 02 17:56:08.379832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.379634 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" event={"ID":"be6a1e76-95ae-4ef4-9123-b2518eee56a0","Type":"ContainerDied","Data":"de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957"} Nov 02 17:56:08.379832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.379697 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b" event={"ID":"be6a1e76-95ae-4ef4-9123-b2518eee56a0","Type":"ContainerDied","Data":"e51ec531812a819fa938bf3bb9701fa69ca698cebd49df504c4f3d7fa1a6f582"} Nov 02 17:56:08.379832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.379750 4310 scope.go:122] "RemoveContainer" containerID="de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957" Nov 02 17:56:08.381198 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.381178 4310 generic.go:356] "Generic (PLEG): container finished" podID="74414d9b-bc66-443d-89b6-03901d5b7890" containerID="64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc" exitCode=0 Nov 02 17:56:08.381277 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.381221 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" Nov 02 17:56:08.381333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.381299 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" event={"ID":"74414d9b-bc66-443d-89b6-03901d5b7890","Type":"ContainerDied","Data":"64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc"} Nov 02 17:56:08.381389 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.381343 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn" event={"ID":"74414d9b-bc66-443d-89b6-03901d5b7890","Type":"ContainerDied","Data":"c0a104fb60432d164a29a9fb7c11f42bd698dc819a34ef76b131904771bede67"} Nov 02 17:56:08.383001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.382979 4310 generic.go:356] "Generic (PLEG): container finished" podID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerID="0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5" exitCode=0 Nov 02 17:56:08.383167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.383090 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" event={"ID":"1801c5bf-b709-48b0-9e6b-721f85eb4520","Type":"ContainerDied","Data":"0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5"} Nov 02 17:56:08.383167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.383130 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" event={"ID":"1801c5bf-b709-48b0-9e6b-721f85eb4520","Type":"ContainerDied","Data":"ed6923ab65f948a3bbeeb95c3713cb2f7437a20b9c50b15733f88210cb9c6cba"} Nov 02 17:56:08.383167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.383147 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc" Nov 02 17:56:08.394127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.394067 4310 scope.go:122] "RemoveContainer" containerID="0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c" Nov 02 17:56:08.404699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.404678 4310 scope.go:122] "RemoveContainer" containerID="de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957" Nov 02 17:56:08.404981 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:56:08.404953 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957\": container with ID starting with de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957 not found: ID does not exist" containerID="de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957" Nov 02 17:56:08.405067 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.404990 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957"} err="failed to get container status \"de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957\": rpc error: code = NotFound desc = could not find container \"de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957\": container with ID starting with de6cfccb4314a51032c6391e370241ff82f129e8471ea4d2d84b699b3fcc3957 not found: ID does not exist" Nov 02 17:56:08.405067 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.405010 4310 scope.go:122] "RemoveContainer" containerID="0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c" Nov 02 17:56:08.405970 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:56:08.405946 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c\": container with ID starting with 0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c not found: ID does not exist" containerID="0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c" Nov 02 17:56:08.406053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.405979 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c"} err="failed to get container status \"0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c\": rpc error: code = NotFound desc = could not find container \"0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c\": container with ID starting with 0a1e8049fcb0324249076522b552f8854ccb08c9f67ffeb6645acdb6b8ab682c not found: ID does not exist" Nov 02 17:56:08.406053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.406000 4310 scope.go:122] "RemoveContainer" containerID="64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc" Nov 02 17:56:08.412946 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.412927 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b"] Nov 02 17:56:08.417561 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.417542 4310 scope.go:122] "RemoveContainer" containerID="64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc" Nov 02 17:56:08.417838 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:56:08.417818 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc\": container with ID starting with 64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc not found: ID does not exist" containerID="64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc" Nov 02 17:56:08.417910 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.417845 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc"} err="failed to get container status \"64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc\": rpc error: code = NotFound desc = could not find container \"64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc\": container with ID starting with 64ef409243bb281a3211d695de04786f56bcd937503abfe20912151a8dfe40bc not found: ID does not exist" Nov 02 17:56:08.417910 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.417861 4310 scope.go:122] "RemoveContainer" containerID="0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5" Nov 02 17:56:08.419585 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.419567 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-82r4b"] Nov 02 17:56:08.432270 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.432245 4310 scope.go:122] "RemoveContainer" containerID="0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5" Nov 02 17:56:08.432711 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:56:08.432673 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5\": container with ID starting with 0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5 not found: ID does not exist" containerID="0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5" Nov 02 17:56:08.432824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.432715 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5"} err="failed to get container status \"0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5\": rpc error: code = NotFound desc = could not find container \"0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5\": container with ID starting with 0122649308aeb563b2875e1fcd18aaa1a8e06e4ae73d411ba634a62178b905b5 not found: ID does not exist" Nov 02 17:56:08.439195 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.439173 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn"] Nov 02 17:56:08.444867 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.444845 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-lhdtn"] Nov 02 17:56:08.452724 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.452695 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc"] Nov 02 17:56:08.458259 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.458241 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7b0683-openshift-default-67fb668965-khnvc"] Nov 02 17:56:08.675393 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.675307 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" path="/var/lib/kubelet/pods/1801c5bf-b709-48b0-9e6b-721f85eb4520/volumes" Nov 02 17:56:08.676020 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.675997 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" path="/var/lib/kubelet/pods/74414d9b-bc66-443d-89b6-03901d5b7890/volumes" Nov 02 17:56:08.676566 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.676545 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" path="/var/lib/kubelet/pods/be6a1e76-95ae-4ef4-9123-b2518eee56a0/volumes" Nov 02 17:56:08.966843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:08.966797 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:09.668983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:09.668541 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:09.860525 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:09.860492 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:10.380981 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:10.380946 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:11.071874 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:11.071835 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:12.156115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:12.156054 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:13.104105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:13.100313 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx"] Nov 02 17:56:13.121110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:13.117856 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6"] Nov 02 17:56:13.183996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:13.168207 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k"] Nov 02 17:56:13.216223 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:13.216136 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a098faa-nkdx6"] Nov 02 17:56:13.227347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:13.227293 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7"] Nov 02 17:56:13.272583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:13.268114 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f2db06-rgc9k"] Nov 02 17:56:13.278227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:13.278172 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0014a1-r7pxx"] Nov 02 17:56:13.289188 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:13.288533 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:13.290124 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:13.290051 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20638c17-nh2q7"] Nov 02 17:56:13.557743 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:13.557712 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:13.868758 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:13.868666 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:14.275645 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:14.275605 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:14.620979 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:14.616449 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:14.677711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:14.677676 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1e6e49-7e35-40b7-aa82-75b609ea5eba" path="/var/lib/kubelet/pods/1f1e6e49-7e35-40b7-aa82-75b609ea5eba/volumes" Nov 02 17:56:14.678262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:14.678236 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43996d70-c3d6-453f-bc7a-67248670d592" path="/var/lib/kubelet/pods/43996d70-c3d6-453f-bc7a-67248670d592/volumes" Nov 02 17:56:14.678713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:14.678698 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5472ba-0efd-47e8-921e-8906e56d4155" path="/var/lib/kubelet/pods/9e5472ba-0efd-47e8-921e-8906e56d4155/volumes" Nov 02 17:56:14.679483 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:14.679467 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc447024-dc18-409b-af2a-de23bb927c21" path="/var/lib/kubelet/pods/dc447024-dc18-409b-af2a-de23bb927c21/volumes" Nov 02 17:56:14.902173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:14.902048 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:15.065736 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:15.065705 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:15.685190 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:15.685150 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:16.345101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:16.344863 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:16.726173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:16.726140 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:17.601148 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:17.601063 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:18.269228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.269187 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:18.539731 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.539657 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf"] Nov 02 17:56:18.540389 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540373 4310 cpu_manager.go:414] "containerMap: removing container" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" Nov 02 17:56:18.540439 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540391 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" Nov 02 17:56:18.540439 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540421 4310 cpu_manager.go:414] "containerMap: removing container" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" Nov 02 17:56:18.540439 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540426 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" Nov 02 17:56:18.540531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540439 4310 cpu_manager.go:414] "containerMap: removing container" podUID="acd6a9c0-5175-4140-9682-6d369c7b7825" containerName="revoke-keys" Nov 02 17:56:18.540531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540445 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd6a9c0-5175-4140-9682-6d369c7b7825" containerName="revoke-keys" Nov 02 17:56:18.540531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540459 4310 cpu_manager.go:414] "containerMap: removing container" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" Nov 02 17:56:18.540531 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540464 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" Nov 02 17:56:18.540651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540564 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" Nov 02 17:56:18.540651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540574 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd6a9c0-5175-4140-9682-6d369c7b7825" containerName="revoke-keys" Nov 02 17:56:18.540651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540581 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" Nov 02 17:56:18.540651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540588 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="74414d9b-bc66-443d-89b6-03901d5b7890" containerName="istio-proxy" Nov 02 17:56:18.540651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.540595 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="1801c5bf-b709-48b0-9e6b-721f85eb4520" containerName="istio-proxy" Nov 02 17:56:18.551038 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.550984 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:18.563226 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.563194 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44475e01-d039-48b4-9120-a451e533076d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:18.563427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.563403 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:56:18.563770 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.563601 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztlz\" (UniqueName: \"kubernetes.io/projected/44475e01-d039-48b4-9120-a451e533076d-kube-api-access-xztlz\") pod \"maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:18.563877 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.563848 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44475e01-d039-48b4-9120-a451e533076d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:18.564389 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.564367 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:56:18.580357 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.580328 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf"] Nov 02 17:56:18.665249 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.665214 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44475e01-d039-48b4-9120-a451e533076d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:18.665435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.665383 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztlz\" (UniqueName: \"kubernetes.io/projected/44475e01-d039-48b4-9120-a451e533076d-kube-api-access-xztlz\") pod \"maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:18.665435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.665407 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44475e01-d039-48b4-9120-a451e533076d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:18.666167 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.666127 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44475e01-d039-48b4-9120-a451e533076d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:18.668655 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.668279 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44475e01-d039-48b4-9120-a451e533076d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:18.675659 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.675634 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztlz\" (UniqueName: \"kubernetes.io/projected/44475e01-d039-48b4-9120-a451e533076d-kube-api-access-xztlz\") pod \"maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:18.882399 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:18.882327 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:19.168612 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:56:19.168582 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44475e01_d039_48b4_9120_a451e533076d.slice/crio-193434e5c5f09d8b4fc4395e597ef21f303ad4c3e61fb7591b3d7258ce3c0005 WatchSource:0}: Error finding container 193434e5c5f09d8b4fc4395e597ef21f303ad4c3e61fb7591b3d7258ce3c0005: Status 404 returned error can't find the container with id 193434e5c5f09d8b4fc4395e597ef21f303ad4c3e61fb7591b3d7258ce3c0005 Nov 02 17:56:19.172160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:19.172142 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf"] Nov 02 17:56:19.180990 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:19.180963 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:19.585416 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:19.585382 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" event={"ID":"44475e01-d039-48b4-9120-a451e533076d","Type":"ContainerStarted","Data":"595014f621ca1b210bf38da9c0d5766efd8251f8888fdfbbf0dfa69c45b22536"} Nov 02 17:56:19.585811 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:19.585434 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" event={"ID":"44475e01-d039-48b4-9120-a451e533076d","Type":"ContainerStarted","Data":"193434e5c5f09d8b4fc4395e597ef21f303ad4c3e61fb7591b3d7258ce3c0005"} Nov 02 17:56:19.607448 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:19.607403 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" podStartSLOduration=1.607389138 podStartE2EDuration="1.607389138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:19.603038428 +0000 UTC m=+977.046121065" watchObservedRunningTime="2026-11-02 17:56:19.607389138 +0000 UTC m=+977.050471764" Nov 02 17:56:19.942108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:19.940531 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:20.595903 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:20.595866 4310 generic.go:356] "Generic (PLEG): container finished" podID="44475e01-d039-48b4-9120-a451e533076d" containerID="595014f621ca1b210bf38da9c0d5766efd8251f8888fdfbbf0dfa69c45b22536" exitCode=0 Nov 02 17:56:20.596290 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:20.595949 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" event={"ID":"44475e01-d039-48b4-9120-a451e533076d","Type":"ContainerDied","Data":"595014f621ca1b210bf38da9c0d5766efd8251f8888fdfbbf0dfa69c45b22536"} Nov 02 17:56:20.654760 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:20.654723 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:21.009365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.009328 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw"] Nov 02 17:56:21.010111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.010092 4310 cpu_manager.go:414] "containerMap: removing container" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" Nov 02 17:56:21.010111 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.010111 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6a1e76-95ae-4ef4-9123-b2518eee56a0" containerName="istio-proxy" Nov 02 17:56:21.014266 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.014249 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:21.022835 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.022810 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw"] Nov 02 17:56:21.037681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.037655 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm"] Nov 02 17:56:21.049537 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.049508 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc3589c-9dbmm"] Nov 02 17:56:21.144144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.144051 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:21.144340 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.144301 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4swpb\" (UniqueName: \"kubernetes.io/projected/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-kube-api-access-4swpb\") pod \"maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:21.144519 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.144502 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:21.245829 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.245793 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:21.246004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.245943 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4swpb\" (UniqueName: \"kubernetes.io/projected/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-kube-api-access-4swpb\") pod \"maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:21.246178 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.246160 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:21.246551 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.246529 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:21.248553 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.248524 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:21.254872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.254850 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4swpb\" (UniqueName: \"kubernetes.io/projected/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-kube-api-access-4swpb\") pod \"maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:21.336296 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.336215 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:21.535804 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.535771 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:21.733211 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.733171 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw"] Nov 02 17:56:21.906070 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.906047 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:21.961251 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.961163 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44475e01-d039-48b4-9120-a451e533076d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44475e01-d039-48b4-9120-a451e533076d-maas-api-xks-ca\") pod \"44475e01-d039-48b4-9120-a451e533076d\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " Nov 02 17:56:21.961462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.961446 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44475e01-d039-48b4-9120-a451e533076d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44475e01-d039-48b4-9120-a451e533076d-maas-api-openshift-ca\") pod \"44475e01-d039-48b4-9120-a451e533076d\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " Nov 02 17:56:21.961551 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.961479 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44475e01-d039-48b4-9120-a451e533076d-kube-api-access-xztlz\" (UniqueName: \"kubernetes.io/projected/44475e01-d039-48b4-9120-a451e533076d-kube-api-access-xztlz\") pod \"44475e01-d039-48b4-9120-a451e533076d\" (UID: \"44475e01-d039-48b4-9120-a451e533076d\") " Nov 02 17:56:21.961908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.961881 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44475e01-d039-48b4-9120-a451e533076d-maas-api-openshift-ca" pod "44475e01-d039-48b4-9120-a451e533076d" (UID: "44475e01-d039-48b4-9120-a451e533076d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:21.962095 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.962061 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44475e01-d039-48b4-9120-a451e533076d-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:21.963747 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.963715 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44475e01-d039-48b4-9120-a451e533076d-kube-api-access-xztlz" pod "44475e01-d039-48b4-9120-a451e533076d" (UID: "44475e01-d039-48b4-9120-a451e533076d"). InnerVolumeSpecName "kube-api-access-xztlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:21.964028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:21.964004 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44475e01-d039-48b4-9120-a451e533076d-maas-api-xks-ca" pod "44475e01-d039-48b4-9120-a451e533076d" (UID: "44475e01-d039-48b4-9120-a451e533076d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:22.026431 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.026392 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg"] Nov 02 17:56:22.032516 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.032490 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af579afc-f5dzg"] Nov 02 17:56:22.064343 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.064311 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xztlz\" (UniqueName: \"kubernetes.io/projected/44475e01-d039-48b4-9120-a451e533076d-kube-api-access-xztlz\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:22.064481 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.064345 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44475e01-d039-48b4-9120-a451e533076d-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:22.346427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.346397 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:22.685608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.685521 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b78e3cb-4752-4ac9-a3f9-066179814979" path="/var/lib/kubelet/pods/0b78e3cb-4752-4ac9-a3f9-066179814979/volumes" Nov 02 17:56:22.686180 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.686157 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860782be-4a11-45ba-9515-ac67ae07091a" path="/var/lib/kubelet/pods/860782be-4a11-45ba-9515-ac67ae07091a/volumes" Nov 02 17:56:22.749736 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.749692 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" event={"ID":"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab","Type":"ContainerStarted","Data":"0cfe717ef10377f9eaae036686b70b5e2790b87b591ec2d44b5078d144916f23"} Nov 02 17:56:22.750356 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.749767 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" event={"ID":"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab","Type":"ContainerStarted","Data":"76a0480e9dbb3779f43f7ac006c475a10f27fefff0a7c486319a2741223d38f0"} Nov 02 17:56:22.751687 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.751653 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" event={"ID":"44475e01-d039-48b4-9120-a451e533076d","Type":"ContainerDied","Data":"193434e5c5f09d8b4fc4395e597ef21f303ad4c3e61fb7591b3d7258ce3c0005"} Nov 02 17:56:22.751687 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.751677 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf" Nov 02 17:56:22.751899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.751713 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="193434e5c5f09d8b4fc4395e597ef21f303ad4c3e61fb7591b3d7258ce3c0005" Nov 02 17:56:22.778763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:22.778709 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" podStartSLOduration=2.778691104 podStartE2EDuration="2.778691104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:22.768037063 +0000 UTC m=+980.211119743" watchObservedRunningTime="2026-11-02 17:56:22.778691104 +0000 UTC m=+980.221773743" Nov 02 17:56:23.204953 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:23.204907 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:23.764737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:23.764703 4310 generic.go:356] "Generic (PLEG): container finished" podID="1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab" containerID="0cfe717ef10377f9eaae036686b70b5e2790b87b591ec2d44b5078d144916f23" exitCode=0 Nov 02 17:56:23.765241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:23.764793 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" event={"ID":"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab","Type":"ContainerDied","Data":"0cfe717ef10377f9eaae036686b70b5e2790b87b591ec2d44b5078d144916f23"} Nov 02 17:56:23.913179 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:23.913136 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:24.581759 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:24.581727 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:24.944706 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:24.944682 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:25.019763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.019732 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-kube-api-access-4swpb\" (UniqueName: \"kubernetes.io/projected/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-kube-api-access-4swpb\") pod \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " Nov 02 17:56:25.019960 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.019899 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-openshift-ca\") pod \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " Nov 02 17:56:25.020157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.020131 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-xks-ca\") pod \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\" (UID: \"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab\") " Nov 02 17:56:25.020370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.020340 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-openshift-ca" pod "1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab" (UID: "1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:25.020752 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.020729 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:25.022310 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.022288 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-xks-ca" pod "1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab" (UID: "1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:25.022771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.022749 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-kube-api-access-4swpb" pod "1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab" (UID: "1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab"). InnerVolumeSpecName "kube-api-access-4swpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:25.122637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.122562 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:25.122637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.122590 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4swpb\" (UniqueName: \"kubernetes.io/projected/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab-kube-api-access-4swpb\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:25.263275 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.263244 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:25.796073 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.796047 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" Nov 02 17:56:25.796073 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.796056 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw" event={"ID":"1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab","Type":"ContainerDied","Data":"76a0480e9dbb3779f43f7ac006c475a10f27fefff0a7c486319a2741223d38f0"} Nov 02 17:56:25.796312 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.796113 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76a0480e9dbb3779f43f7ac006c475a10f27fefff0a7c486319a2741223d38f0" Nov 02 17:56:25.964014 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:25.963972 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:26.102170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:26.093583 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw"] Nov 02 17:56:26.117637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:26.117584 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a415d5cf-6btnw"] Nov 02 17:56:26.570655 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:26.570625 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:26.730818 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:26.730764 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28aaacdf-f1cf-407f-92f8-eba32640d71f" path="/var/lib/kubelet/pods/28aaacdf-f1cf-407f-92f8-eba32640d71f/volumes" Nov 02 17:56:27.241435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:27.241401 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:27.854369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:27.854337 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:28.044048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:28.044012 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv"] Nov 02 17:56:28.053359 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:28.053330 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4717896-vbrnv"] Nov 02 17:56:28.665035 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:28.665003 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:28.691151 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:28.691117 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390b7a47-77c3-4a9f-8bc8-ed6735f823ac" path="/var/lib/kubelet/pods/390b7a47-77c3-4a9f-8bc8-ed6735f823ac/volumes" Nov 02 17:56:28.766140 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:28.766011 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:29.091183 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:29.091142 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86"] Nov 02 17:56:29.123099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:29.117144 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0b5bab-t7z86"] Nov 02 17:56:29.557105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:29.555776 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:30.155321 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:30.155266 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:30.679295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:30.679262 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ee8219-2afe-469b-abaa-da3305f86cf2" path="/var/lib/kubelet/pods/77ee8219-2afe-469b-abaa-da3305f86cf2/volumes" Nov 02 17:56:30.945173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:30.942636 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:31.170258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:31.170223 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42"] Nov 02 17:56:31.234407 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:31.234354 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:31.345380 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:31.345345 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:31.851301 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:31.851271 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:32.486515 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:32.486472 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:33.026826 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:33.026795 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn"] Nov 02 17:56:33.032980 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:33.032953 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv"] Nov 02 17:56:33.039304 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:33.039259 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5d5ac3-b2bvn"] Nov 02 17:56:33.044751 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:33.044725 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55ad14ed-cfjvv"] Nov 02 17:56:33.280353 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:33.280256 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:33.525759 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:56:33.525726 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:56:33.850129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:33.850098 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:34.031742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:34.031575 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:34.365932 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:34.365886 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf"] Nov 02 17:56:34.373448 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:34.373420 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e86b86-v6mpf"] Nov 02 17:56:34.386879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:34.386806 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:34.524831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:34.524795 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:34.690997 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:34.690900 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266551f9-492e-4113-ac11-5f7fc6ba0304" path="/var/lib/kubelet/pods/266551f9-492e-4113-ac11-5f7fc6ba0304/volumes" Nov 02 17:56:34.691720 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:34.691695 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8466b8-2d52-4b4a-85a7-a566b8ad691d" path="/var/lib/kubelet/pods/2a8466b8-2d52-4b4a-85a7-a566b8ad691d/volumes" Nov 02 17:56:34.693070 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:34.693046 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44475e01-d039-48b4-9120-a451e533076d" path="/var/lib/kubelet/pods/44475e01-d039-48b4-9120-a451e533076d/volumes" Nov 02 17:56:35.034500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:35.034470 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn"] Nov 02 17:56:35.040412 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:35.040388 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0324a29-hdzdn"] Nov 02 17:56:35.301653 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:35.301572 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:35.918026 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:35.917991 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:36.223204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:36.223155 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" podUID="c97b6368-9d5a-4eea-bc51-38db653b0396" containerName="payload-pre-processing" containerID="cri-o://42268310903d7fb1417c00a4310cc85203f1058192fe0096d85456ebce116c76" gracePeriod=25 Nov 02 17:56:36.224624 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:36.224589 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:36.384136 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:36.381675 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:36.517226 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:36.515689 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:36.762127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:36.761436 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb63e790-c7bc-4312-9afb-30049d3bb405" path="/var/lib/kubelet/pods/fb63e790-c7bc-4312-9afb-30049d3bb405/volumes" Nov 02 17:56:36.932923 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:36.932438 4310 generic.go:356] "Generic (PLEG): container finished" podID="c97b6368-9d5a-4eea-bc51-38db653b0396" containerID="42268310903d7fb1417c00a4310cc85203f1058192fe0096d85456ebce116c76" exitCode=0 Nov 02 17:56:36.932923 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:36.932650 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" event={"ID":"c97b6368-9d5a-4eea-bc51-38db653b0396","Type":"ContainerDied","Data":"42268310903d7fb1417c00a4310cc85203f1058192fe0096d85456ebce116c76"} Nov 02 17:56:36.985353 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:36.985321 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:37.112108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.092148 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw"] Nov 02 17:56:37.206409 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.206147 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:37.230065 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.227038 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:37.254738 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.246736 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-33a0a14d-rt8mw"] Nov 02 17:56:37.254988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.254965 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p"] Nov 02 17:56:37.280121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.272812 4310 cpu_manager.go:414] "containerMap: removing container" podUID="1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab" containerName="revoke-keys" Nov 02 17:56:37.280121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.272843 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab" containerName="revoke-keys" Nov 02 17:56:37.280121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.272895 4310 cpu_manager.go:414] "containerMap: removing container" podUID="44475e01-d039-48b4-9120-a451e533076d" containerName="revoke-keys" Nov 02 17:56:37.280121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.272904 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44475e01-d039-48b4-9120-a451e533076d" containerName="revoke-keys" Nov 02 17:56:37.280121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.272951 4310 cpu_manager.go:414] "containerMap: removing container" podUID="c97b6368-9d5a-4eea-bc51-38db653b0396" containerName="payload-pre-processing" Nov 02 17:56:37.280121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.272960 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97b6368-9d5a-4eea-bc51-38db653b0396" containerName="payload-pre-processing" Nov 02 17:56:37.280121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.273374 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab" containerName="revoke-keys" Nov 02 17:56:37.280121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.273408 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="44475e01-d039-48b4-9120-a451e533076d" containerName="revoke-keys" Nov 02 17:56:37.280121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.273421 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97b6368-9d5a-4eea-bc51-38db653b0396" containerName="payload-pre-processing" Nov 02 17:56:37.303192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.298168 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p"] Nov 02 17:56:37.303192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.298299 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:37.342679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.342647 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:37.343776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.343751 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97b6368-9d5a-4eea-bc51-38db653b0396-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97b6368-9d5a-4eea-bc51-38db653b0396-plugins-config-volume\") pod \"c97b6368-9d5a-4eea-bc51-38db653b0396\" (UID: \"c97b6368-9d5a-4eea-bc51-38db653b0396\") " Nov 02 17:56:37.344066 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.344047 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97b6368-9d5a-4eea-bc51-38db653b0396-kube-api-access-fx8hb\" (UniqueName: \"kubernetes.io/projected/c97b6368-9d5a-4eea-bc51-38db653b0396-kube-api-access-fx8hb\") pod \"c97b6368-9d5a-4eea-bc51-38db653b0396\" (UID: \"c97b6368-9d5a-4eea-bc51-38db653b0396\") " Nov 02 17:56:37.346259 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.346233 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97b6368-9d5a-4eea-bc51-38db653b0396-plugins-config-volume" pod "c97b6368-9d5a-4eea-bc51-38db653b0396" (UID: "c97b6368-9d5a-4eea-bc51-38db653b0396"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:37.376987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.363191 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:56:37.376987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.363302 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:56:37.376987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.363951 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97b6368-9d5a-4eea-bc51-38db653b0396-kube-api-access-fx8hb" pod "c97b6368-9d5a-4eea-bc51-38db653b0396" (UID: "c97b6368-9d5a-4eea-bc51-38db653b0396"). InnerVolumeSpecName "kube-api-access-fx8hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:37.453130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.450781 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:37.453130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.450955 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7gt7\" (UniqueName: \"kubernetes.io/projected/163da1e2-9ee5-49d0-86ea-4069654a73f4-kube-api-access-b7gt7\") pod \"maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:37.453130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.451305 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:37.453130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.451468 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97b6368-9d5a-4eea-bc51-38db653b0396-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:37.453130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.451487 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx8hb\" (UniqueName: \"kubernetes.io/projected/c97b6368-9d5a-4eea-bc51-38db653b0396-kube-api-access-fx8hb\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:37.590113 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.556188 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:37.590113 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.556284 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:37.590113 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.556428 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7gt7\" (UniqueName: \"kubernetes.io/projected/163da1e2-9ee5-49d0-86ea-4069654a73f4-kube-api-access-b7gt7\") pod \"maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:37.590113 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.557823 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:37.590113 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.578535 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7gt7\" (UniqueName: \"kubernetes.io/projected/163da1e2-9ee5-49d0-86ea-4069654a73f4-kube-api-access-b7gt7\") pod \"maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:37.606113 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.595738 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:37.658727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.658690 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:37.662371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.662323 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:37.805090 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.805046 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp"] Nov 02 17:56:37.809872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.809852 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:37.821104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.821065 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp"] Nov 02 17:56:37.861726 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.861650 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:37.861879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.861867 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87c5f\" (UniqueName: \"kubernetes.io/projected/991b4d35-a453-4122-b3d4-ab356e3304f2-kube-api-access-87c5f\") pod \"maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:37.861919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.861895 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:37.944553 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.944522 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" event={"ID":"c97b6368-9d5a-4eea-bc51-38db653b0396","Type":"ContainerDied","Data":"f89fdcc99a2655a8e6e0918dfe4de636a13c0731979c4b558ba2944d1069d5c8"} Nov 02 17:56:37.944934 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.944581 4310 scope.go:122] "RemoveContainer" containerID="42268310903d7fb1417c00a4310cc85203f1058192fe0096d85456ebce116c76" Nov 02 17:56:37.944934 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.944716 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42" Nov 02 17:56:37.964556 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.964127 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87c5f\" (UniqueName: \"kubernetes.io/projected/991b4d35-a453-4122-b3d4-ab356e3304f2-kube-api-access-87c5f\") pod \"maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:37.964556 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.964177 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:37.964556 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.964350 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:37.965064 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.965040 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:37.966675 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.966647 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:37.966902 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:56:37.966880 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163da1e2_9ee5_49d0_86ea_4069654a73f4.slice/crio-67f7c3b59d7c14805f112790b89c40fc7d282ca93e39abf8c0ea4205f337d7ce WatchSource:0}: Error finding container 67f7c3b59d7c14805f112790b89c40fc7d282ca93e39abf8c0ea4205f337d7ce: Status 404 returned error can't find the container with id 67f7c3b59d7c14805f112790b89c40fc7d282ca93e39abf8c0ea4205f337d7ce Nov 02 17:56:37.968755 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.968540 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p"] Nov 02 17:56:37.978790 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.978766 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87c5f\" (UniqueName: \"kubernetes.io/projected/991b4d35-a453-4122-b3d4-ab356e3304f2-kube-api-access-87c5f\") pod \"maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:37.978956 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.978938 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42"] Nov 02 17:56:37.988528 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:37.988505 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-33a0a14d-6c8fcb5555-zlk42"] Nov 02 17:56:38.131718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:38.131627 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:38.423546 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:56:38.423516 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991b4d35_a453_4122_b3d4_ab356e3304f2.slice/crio-63c12bd6e861adaf493f3b6ade78a0240c5a446a59a4760baf29c8dce2834fb7 WatchSource:0}: Error finding container 63c12bd6e861adaf493f3b6ade78a0240c5a446a59a4760baf29c8dce2834fb7: Status 404 returned error can't find the container with id 63c12bd6e861adaf493f3b6ade78a0240c5a446a59a4760baf29c8dce2834fb7 Nov 02 17:56:38.425940 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:38.425915 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp"] Nov 02 17:56:38.539752 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:38.539717 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:38.726592 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:38.726559 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab" path="/var/lib/kubelet/pods/1d25e26d-1a6f-4cb5-a052-a1dbdc49b2ab/volumes" Nov 02 17:56:38.727138 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:38.727118 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97b6368-9d5a-4eea-bc51-38db653b0396" path="/var/lib/kubelet/pods/c97b6368-9d5a-4eea-bc51-38db653b0396/volumes" Nov 02 17:56:38.960727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:38.960688 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" event={"ID":"991b4d35-a453-4122-b3d4-ab356e3304f2","Type":"ContainerStarted","Data":"1b52e01cda912fad94a201d6c8968cdfcf702c6431f58f95d96622a0e116c460"} Nov 02 17:56:38.961164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:38.960763 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" event={"ID":"991b4d35-a453-4122-b3d4-ab356e3304f2","Type":"ContainerStarted","Data":"63c12bd6e861adaf493f3b6ade78a0240c5a446a59a4760baf29c8dce2834fb7"} Nov 02 17:56:38.963616 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:38.963589 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" event={"ID":"163da1e2-9ee5-49d0-86ea-4069654a73f4","Type":"ContainerStarted","Data":"04f3ab1bb73daf630578e517a7ad1e09dd6979f3a58ab043d2e45edc48c8960c"} Nov 02 17:56:38.963754 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:38.963632 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" event={"ID":"163da1e2-9ee5-49d0-86ea-4069654a73f4","Type":"ContainerStarted","Data":"67f7c3b59d7c14805f112790b89c40fc7d282ca93e39abf8c0ea4205f337d7ce"} Nov 02 17:56:38.981506 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:38.981458 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" podStartSLOduration=1.981442763 podStartE2EDuration="1.981442763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:38.978702845 +0000 UTC m=+996.421785481" watchObservedRunningTime="2026-11-02 17:56:38.981442763 +0000 UTC m=+996.424525402" Nov 02 17:56:38.997894 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:38.997853 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" podStartSLOduration=1.997840614 podStartE2EDuration="1.997840614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:38.994548209 +0000 UTC m=+996.437630840" watchObservedRunningTime="2026-11-02 17:56:38.997840614 +0000 UTC m=+996.440923268" Nov 02 17:56:39.194154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:39.194055 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:39.933102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:39.932265 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:39.997114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:39.996667 4310 generic.go:356] "Generic (PLEG): container finished" podID="991b4d35-a453-4122-b3d4-ab356e3304f2" containerID="1b52e01cda912fad94a201d6c8968cdfcf702c6431f58f95d96622a0e116c460" exitCode=0 Nov 02 17:56:39.997114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:39.996841 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" event={"ID":"991b4d35-a453-4122-b3d4-ab356e3304f2","Type":"ContainerDied","Data":"1b52e01cda912fad94a201d6c8968cdfcf702c6431f58f95d96622a0e116c460"} Nov 02 17:56:40.005345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:40.000132 4310 generic.go:356] "Generic (PLEG): container finished" podID="163da1e2-9ee5-49d0-86ea-4069654a73f4" containerID="04f3ab1bb73daf630578e517a7ad1e09dd6979f3a58ab043d2e45edc48c8960c" exitCode=0 Nov 02 17:56:40.005345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:40.000221 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" event={"ID":"163da1e2-9ee5-49d0-86ea-4069654a73f4","Type":"ContainerDied","Data":"04f3ab1bb73daf630578e517a7ad1e09dd6979f3a58ab043d2e45edc48c8960c"} Nov 02 17:56:40.625452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:40.625415 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:41.195997 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.195969 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:41.246938 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.246903 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:41.344549 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.344460 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf"] Nov 02 17:56:41.358787 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.358757 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9"] Nov 02 17:56:41.362018 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.361984 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-xks-ca\") pod \"163da1e2-9ee5-49d0-86ea-4069654a73f4\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " Nov 02 17:56:41.362243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.362224 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163da1e2-9ee5-49d0-86ea-4069654a73f4-kube-api-access-b7gt7\" (UniqueName: \"kubernetes.io/projected/163da1e2-9ee5-49d0-86ea-4069654a73f4-kube-api-access-b7gt7\") pod \"163da1e2-9ee5-49d0-86ea-4069654a73f4\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " Nov 02 17:56:41.362324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.362291 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-openshift-ca\") pod \"163da1e2-9ee5-49d0-86ea-4069654a73f4\" (UID: \"163da1e2-9ee5-49d0-86ea-4069654a73f4\") " Nov 02 17:56:41.362380 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.362339 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-xks-ca\") pod \"991b4d35-a453-4122-b3d4-ab356e3304f2\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " Nov 02 17:56:41.362520 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.362417 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-openshift-ca\") pod \"991b4d35-a453-4122-b3d4-ab356e3304f2\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " Nov 02 17:56:41.362817 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.362792 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991b4d35-a453-4122-b3d4-ab356e3304f2-kube-api-access-87c5f\" (UniqueName: \"kubernetes.io/projected/991b4d35-a453-4122-b3d4-ab356e3304f2-kube-api-access-87c5f\") pod \"991b4d35-a453-4122-b3d4-ab356e3304f2\" (UID: \"991b4d35-a453-4122-b3d4-ab356e3304f2\") " Nov 02 17:56:41.363512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.363483 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-openshift-ca" pod "991b4d35-a453-4122-b3d4-ab356e3304f2" (UID: "991b4d35-a453-4122-b3d4-ab356e3304f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:41.363823 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.363799 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:41.363979 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.363956 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-openshift-ca" pod "163da1e2-9ee5-49d0-86ea-4069654a73f4" (UID: "163da1e2-9ee5-49d0-86ea-4069654a73f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:41.365364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.365324 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991b4d35-a453-4122-b3d4-ab356e3304f2-kube-api-access-87c5f" pod "991b4d35-a453-4122-b3d4-ab356e3304f2" (UID: "991b4d35-a453-4122-b3d4-ab356e3304f2"). InnerVolumeSpecName "kube-api-access-87c5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:41.366222 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.365961 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-xks-ca" pod "991b4d35-a453-4122-b3d4-ab356e3304f2" (UID: "991b4d35-a453-4122-b3d4-ab356e3304f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:41.366930 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.366895 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163da1e2-9ee5-49d0-86ea-4069654a73f4-kube-api-access-b7gt7" pod "163da1e2-9ee5-49d0-86ea-4069654a73f4" (UID: "163da1e2-9ee5-49d0-86ea-4069654a73f4"). InnerVolumeSpecName "kube-api-access-b7gt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:41.373347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.373151 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-xks-ca" pod "163da1e2-9ee5-49d0-86ea-4069654a73f4" (UID: "163da1e2-9ee5-49d0-86ea-4069654a73f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:41.391986 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.391957 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:41.480654 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.479069 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:41.491313 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.491277 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7gt7\" (UniqueName: \"kubernetes.io/projected/163da1e2-9ee5-49d0-86ea-4069654a73f4-kube-api-access-b7gt7\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:41.491662 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.491638 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163da1e2-9ee5-49d0-86ea-4069654a73f4-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:41.491796 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.491782 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991b4d35-a453-4122-b3d4-ab356e3304f2-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:41.491889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.491877 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87c5f\" (UniqueName: \"kubernetes.io/projected/991b4d35-a453-4122-b3d4-ab356e3304f2-kube-api-access-87c5f\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:41.584645 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:41.584610 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:42.020259 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:42.020229 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" Nov 02 17:56:42.020446 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:42.020232 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp" event={"ID":"991b4d35-a453-4122-b3d4-ab356e3304f2","Type":"ContainerDied","Data":"63c12bd6e861adaf493f3b6ade78a0240c5a446a59a4760baf29c8dce2834fb7"} Nov 02 17:56:42.020446 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:42.020348 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63c12bd6e861adaf493f3b6ade78a0240c5a446a59a4760baf29c8dce2834fb7" Nov 02 17:56:42.021998 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:42.021977 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" Nov 02 17:56:42.021998 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:42.021990 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p" event={"ID":"163da1e2-9ee5-49d0-86ea-4069654a73f4","Type":"ContainerDied","Data":"67f7c3b59d7c14805f112790b89c40fc7d282ca93e39abf8c0ea4205f337d7ce"} Nov 02 17:56:42.022191 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:42.022033 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67f7c3b59d7c14805f112790b89c40fc7d282ca93e39abf8c0ea4205f337d7ce" Nov 02 17:56:42.158713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:42.158681 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:42.626694 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:42.626653 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m"] Nov 02 17:56:42.635210 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:42.635184 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4"] Nov 02 17:56:42.725156 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:42.725119 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:43.072342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:43.072314 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24"] Nov 02 17:56:43.078134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:43.078096 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd412da1-fmj24"] Nov 02 17:56:43.159157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:43.149639 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:43.861336 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:43.861301 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:44.048181 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:44.047270 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx"] Nov 02 17:56:44.055824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:44.055800 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc"] Nov 02 17:56:44.061190 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:44.061166 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9690cd4d-xhxhx"] Nov 02 17:56:44.066310 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:44.066285 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e00247bb-5c6vc"] Nov 02 17:56:44.667779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:44.667739 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:44.677045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:44.677014 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cccdbb63-26f1-4b82-82d2-cf4edb0a499a" path="/var/lib/kubelet/pods/cccdbb63-26f1-4b82-82d2-cf4edb0a499a/volumes" Nov 02 17:56:44.677645 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:44.677627 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d618ff6b-44d2-41cf-af53-86e4f7bf4b3c" path="/var/lib/kubelet/pods/d618ff6b-44d2-41cf-af53-86e4f7bf4b3c/volumes" Nov 02 17:56:44.678059 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:44.678046 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a" path="/var/lib/kubelet/pods/d9a0d902-ffde-4bfb-a5d5-f00fdbdea45a/volumes" Nov 02 17:56:45.492871 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:45.492836 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:46.382909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:46.382870 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:47.037566 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:47.037356 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9"] Nov 02 17:56:47.066101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:47.062967 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720e2077-v8td9"] Nov 02 17:56:47.324245 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:47.324156 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:47.572369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:47.571540 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn"] Nov 02 17:56:47.646513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:47.646429 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52"] Nov 02 17:56:47.817101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:47.797598 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:47.940015 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:47.939908 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:48.247277 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:48.235934 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:48.429998 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:48.429946 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:49.068774 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:49.068033 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:56:49.071367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:49.071324 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:56:49.217131 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:49.214340 4310 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:56:49.226903 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:49.226865 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Nov 02 17:56:49.265099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:49.236429 4310 patch_prober.go:29] interesting pod/console-5b68b9d69f-jlqkc container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.17:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:56:49.265099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:49.236488 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5b68b9d69f-jlqkc" podUID="24c27b5e-11b0-489c-b3cd-6eface16a294" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:56:49.267666 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:49.267633 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3439928-8ca9-4d71-8fe2-8bfb959520ef" path="/var/lib/kubelet/pods/a3439928-8ca9-4d71-8fe2-8bfb959520ef/volumes" Nov 02 17:56:49.336193 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:49.336103 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4"] Nov 02 17:56:49.392201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:49.392158 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:50.301503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.301468 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:50.872877 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.872841 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv"] Nov 02 17:56:50.873571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.873555 4310 cpu_manager.go:414] "containerMap: removing container" podUID="991b4d35-a453-4122-b3d4-ab356e3304f2" containerName="revoke-keys" Nov 02 17:56:50.873571 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.873571 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991b4d35-a453-4122-b3d4-ab356e3304f2" containerName="revoke-keys" Nov 02 17:56:50.873704 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.873637 4310 cpu_manager.go:414] "containerMap: removing container" podUID="163da1e2-9ee5-49d0-86ea-4069654a73f4" containerName="revoke-keys" Nov 02 17:56:50.873704 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.873645 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163da1e2-9ee5-49d0-86ea-4069654a73f4" containerName="revoke-keys" Nov 02 17:56:50.873820 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.873807 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="163da1e2-9ee5-49d0-86ea-4069654a73f4" containerName="revoke-keys" Nov 02 17:56:50.873856 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.873825 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="991b4d35-a453-4122-b3d4-ab356e3304f2" containerName="revoke-keys" Nov 02 17:56:50.877630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.877608 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:50.884733 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.884702 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:56:50.884860 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.884850 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:56:50.887551 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.887532 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77"] Nov 02 17:56:50.897888 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.897862 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv"] Nov 02 17:56:50.898039 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.897982 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:50.906673 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.906645 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77"] Nov 02 17:56:50.946291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.946257 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23336a10-v5k77\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:50.946477 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.946313 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t484\" (UniqueName: \"kubernetes.io/projected/1183c12f-ff3c-4f0c-adf0-a0883c99e839-kube-api-access-4t484\") pod \"maas-api-revoke-sub-23336a10-v5k77\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:50.946477 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.946456 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpqxz\" (UniqueName: \"kubernetes.io/projected/b6f80910-45f6-436c-b553-a41ebe2808ea-kube-api-access-dpqxz\") pod \"maas-api-revoke-sub-05fbb349-fqhfv\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:50.946611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.946495 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23336a10-v5k77\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:50.946700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.946673 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05fbb349-fqhfv\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:50.946766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:50.946734 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05fbb349-fqhfv\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:51.059766 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.059718 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpqxz\" (UniqueName: \"kubernetes.io/projected/b6f80910-45f6-436c-b553-a41ebe2808ea-kube-api-access-dpqxz\") pod \"maas-api-revoke-sub-05fbb349-fqhfv\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:51.059977 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.059784 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23336a10-v5k77\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:51.059977 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.059918 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05fbb349-fqhfv\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:51.060154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.059978 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05fbb349-fqhfv\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:51.060224 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.060205 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23336a10-v5k77\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:51.060285 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.060270 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t484\" (UniqueName: \"kubernetes.io/projected/1183c12f-ff3c-4f0c-adf0-a0883c99e839-kube-api-access-4t484\") pod \"maas-api-revoke-sub-23336a10-v5k77\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:51.061387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.061359 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05fbb349-fqhfv\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:51.061775 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.061745 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23336a10-v5k77\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:51.063241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.063209 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05fbb349-fqhfv\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:51.063448 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.063424 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23336a10-v5k77\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:51.071344 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.071320 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpqxz\" (UniqueName: \"kubernetes.io/projected/b6f80910-45f6-436c-b553-a41ebe2808ea-kube-api-access-dpqxz\") pod \"maas-api-revoke-sub-05fbb349-fqhfv\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:51.078647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.078624 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t484\" (UniqueName: \"kubernetes.io/projected/1183c12f-ff3c-4f0c-adf0-a0883c99e839-kube-api-access-4t484\") pod \"maas-api-revoke-sub-23336a10-v5k77\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:51.198416 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.198340 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:51.208461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.208432 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:51.217554 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.217530 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:51.615091 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:56:51.615039 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f80910_45f6_436c_b553_a41ebe2808ea.slice/crio-baea2321bd9210bef21472e743e9188a4fe03a89fd8058b960234bf5fa74355f WatchSource:0}: Error finding container baea2321bd9210bef21472e743e9188a4fe03a89fd8058b960234bf5fa74355f: Status 404 returned error can't find the container with id baea2321bd9210bef21472e743e9188a4fe03a89fd8058b960234bf5fa74355f Nov 02 17:56:51.616762 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.616684 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv"] Nov 02 17:56:51.642861 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:56:51.642828 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1183c12f_ff3c_4f0c_adf0_a0883c99e839.slice/crio-870b2413a4233fb349c7d243f7d4b00bb4ead12ccd513548f9f84ccbcd3a1a3a WatchSource:0}: Error finding container 870b2413a4233fb349c7d243f7d4b00bb4ead12ccd513548f9f84ccbcd3a1a3a: Status 404 returned error can't find the container with id 870b2413a4233fb349c7d243f7d4b00bb4ead12ccd513548f9f84ccbcd3a1a3a Nov 02 17:56:51.646148 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:51.646053 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77"] Nov 02 17:56:52.254355 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:52.254324 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:52.572241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:52.572197 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" podUID="b453de3f-2c9c-4705-ab30-b3eae093a0b0" containerName="payload-processing" containerID="cri-o://b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a" gracePeriod=25 Nov 02 17:56:52.628132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:52.628063 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" event={"ID":"1183c12f-ff3c-4f0c-adf0-a0883c99e839","Type":"ContainerStarted","Data":"0a0b3918d8b6ad14c50d7ddc205dfe8be91441394c05b17e027c2e86223f8078"} Nov 02 17:56:52.628501 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:52.628155 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" event={"ID":"1183c12f-ff3c-4f0c-adf0-a0883c99e839","Type":"ContainerStarted","Data":"870b2413a4233fb349c7d243f7d4b00bb4ead12ccd513548f9f84ccbcd3a1a3a"} Nov 02 17:56:52.630086 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:52.630049 4310 generic.go:356] "Generic (PLEG): container finished" podID="b6f80910-45f6-436c-b553-a41ebe2808ea" containerID="cf822c6e1272407d38e23ae4aaca99a256f14ab0f3d50ad785c947ec6ddd8d10" exitCode=0 Nov 02 17:56:52.630186 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:52.630130 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" event={"ID":"b6f80910-45f6-436c-b553-a41ebe2808ea","Type":"ContainerDied","Data":"cf822c6e1272407d38e23ae4aaca99a256f14ab0f3d50ad785c947ec6ddd8d10"} Nov 02 17:56:52.630186 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:52.630167 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" event={"ID":"b6f80910-45f6-436c-b553-a41ebe2808ea","Type":"ContainerStarted","Data":"baea2321bd9210bef21472e743e9188a4fe03a89fd8058b960234bf5fa74355f"} Nov 02 17:56:52.647588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:52.647552 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" podUID="2469cdd9-c68f-477b-9cb8-85aadba1bacd" containerName="payload-pre-processing" containerID="cri-o://b57773de42695ac73c95e591f83f2ea173a22f3600d650794563c869c45f92ae" gracePeriod=25 Nov 02 17:56:52.656027 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:52.655987 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" podStartSLOduration=2.655972469 podStartE2EDuration="2.655972469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:52.654339706 +0000 UTC m=+1010.097422344" watchObservedRunningTime="2026-11-02 17:56:52.655972469 +0000 UTC m=+1010.099055102" Nov 02 17:56:53.132524 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.132478 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:53.641234 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.641197 4310 generic.go:356] "Generic (PLEG): container finished" podID="1183c12f-ff3c-4f0c-adf0-a0883c99e839" containerID="0a0b3918d8b6ad14c50d7ddc205dfe8be91441394c05b17e027c2e86223f8078" exitCode=0 Nov 02 17:56:53.641739 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.641291 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" event={"ID":"1183c12f-ff3c-4f0c-adf0-a0883c99e839","Type":"ContainerDied","Data":"0a0b3918d8b6ad14c50d7ddc205dfe8be91441394c05b17e027c2e86223f8078"} Nov 02 17:56:53.662889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.662861 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6"] Nov 02 17:56:53.669456 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.669428 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:53.680792 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.680762 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq"] Nov 02 17:56:53.716010 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.715981 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:53.720415 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.720380 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9sks\" (UniqueName: \"kubernetes.io/projected/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-kube-api-access-w9sks\") pod \"maas-api-revoke-sub-8b9ed352-c9lj6\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:53.721416 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.721391 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9ed352-c9lj6\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:53.721973 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.721848 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9ed352-c9lj6\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:53.755679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.755641 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6"] Nov 02 17:56:53.765172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.765143 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq"] Nov 02 17:56:53.824340 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.824306 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9ed352-c9lj6\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:53.824543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.824468 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b552ade7-lc5lq\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:53.824543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.824502 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b552ade7-lc5lq\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:53.824543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.824537 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9sks\" (UniqueName: \"kubernetes.io/projected/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-kube-api-access-w9sks\") pod \"maas-api-revoke-sub-8b9ed352-c9lj6\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:53.824737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.824567 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kl7r\" (UniqueName: \"kubernetes.io/projected/688fa1f2-1628-404a-9a23-39640c88484c-kube-api-access-7kl7r\") pod \"maas-api-revoke-sub-b552ade7-lc5lq\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:53.824737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.824610 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9ed352-c9lj6\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:53.825378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.825340 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9ed352-c9lj6\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:53.828722 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.828694 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9ed352-c9lj6\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:53.834651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.834625 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9sks\" (UniqueName: \"kubernetes.io/projected/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-kube-api-access-w9sks\") pod \"maas-api-revoke-sub-8b9ed352-c9lj6\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:53.906435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.906367 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:53.926016 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.925979 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b552ade7-lc5lq\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:53.926193 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.926037 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b552ade7-lc5lq\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:53.926193 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.926133 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl7r\" (UniqueName: \"kubernetes.io/projected/688fa1f2-1628-404a-9a23-39640c88484c-kube-api-access-7kl7r\") pod \"maas-api-revoke-sub-b552ade7-lc5lq\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:53.927332 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.927231 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b552ade7-lc5lq\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:53.929923 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.929899 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b552ade7-lc5lq\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:53.936876 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:53.936843 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kl7r\" (UniqueName: \"kubernetes.io/projected/688fa1f2-1628-404a-9a23-39640c88484c-kube-api-access-7kl7r\") pod \"maas-api-revoke-sub-b552ade7-lc5lq\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:54.026452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.026412 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:54.029317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.029274 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-xks-ca\") pod \"b6f80910-45f6-436c-b553-a41ebe2808ea\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " Nov 02 17:56:54.030998 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.030964 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-openshift-ca\") pod \"b6f80910-45f6-436c-b553-a41ebe2808ea\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " Nov 02 17:56:54.031508 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.031488 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f80910-45f6-436c-b553-a41ebe2808ea-kube-api-access-dpqxz\" (UniqueName: \"kubernetes.io/projected/b6f80910-45f6-436c-b553-a41ebe2808ea-kube-api-access-dpqxz\") pod \"b6f80910-45f6-436c-b553-a41ebe2808ea\" (UID: \"b6f80910-45f6-436c-b553-a41ebe2808ea\") " Nov 02 17:56:54.032985 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.032058 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-openshift-ca" pod "b6f80910-45f6-436c-b553-a41ebe2808ea" (UID: "b6f80910-45f6-436c-b553-a41ebe2808ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:54.050042 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.049961 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:54.051845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.051819 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:54.104505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.103958 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f80910-45f6-436c-b553-a41ebe2808ea-kube-api-access-dpqxz" pod "b6f80910-45f6-436c-b553-a41ebe2808ea" (UID: "b6f80910-45f6-436c-b553-a41ebe2808ea"). InnerVolumeSpecName "kube-api-access-dpqxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:54.104505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.104051 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-xks-ca" pod "b6f80910-45f6-436c-b553-a41ebe2808ea" (UID: "b6f80910-45f6-436c-b553-a41ebe2808ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:54.140178 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.140066 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:54.140477 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.140432 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f80910-45f6-436c-b553-a41ebe2808ea-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:54.140477 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.140458 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpqxz\" (UniqueName: \"kubernetes.io/projected/b6f80910-45f6-436c-b553-a41ebe2808ea-kube-api-access-dpqxz\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:54.353415 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.353350 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" podUID="1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a" containerName="payload-pre-processing" containerID="cri-o://badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f" gracePeriod=25 Nov 02 17:56:54.410187 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:56:54.410155 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688fa1f2_1628_404a_9a23_39640c88484c.slice/crio-16f8f945638873cb181c357c5c03cdb92e3bd5ccc9e2c96da3b2ae79308d37e0 WatchSource:0}: Error finding container 16f8f945638873cb181c357c5c03cdb92e3bd5ccc9e2c96da3b2ae79308d37e0: Status 404 returned error can't find the container with id 16f8f945638873cb181c357c5c03cdb92e3bd5ccc9e2c96da3b2ae79308d37e0 Nov 02 17:56:54.412225 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.412144 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq"] Nov 02 17:56:54.435582 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:56:54.435558 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc14d78_414c_48da_8d5e_0828ea8cc5c1.slice/crio-19e6bdfe539950d9ec835083be12fe9f43d7749e2636d10f07026528a6bbb6d3 WatchSource:0}: Error finding container 19e6bdfe539950d9ec835083be12fe9f43d7749e2636d10f07026528a6bbb6d3: Status 404 returned error can't find the container with id 19e6bdfe539950d9ec835083be12fe9f43d7749e2636d10f07026528a6bbb6d3 Nov 02 17:56:54.439696 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.439671 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6"] Nov 02 17:56:54.655402 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.655371 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" event={"ID":"688fa1f2-1628-404a-9a23-39640c88484c","Type":"ContainerStarted","Data":"16f8f945638873cb181c357c5c03cdb92e3bd5ccc9e2c96da3b2ae79308d37e0"} Nov 02 17:56:54.656681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.656661 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" Nov 02 17:56:54.656796 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.656697 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv" event={"ID":"b6f80910-45f6-436c-b553-a41ebe2808ea","Type":"ContainerDied","Data":"baea2321bd9210bef21472e743e9188a4fe03a89fd8058b960234bf5fa74355f"} Nov 02 17:56:54.656796 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.656743 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baea2321bd9210bef21472e743e9188a4fe03a89fd8058b960234bf5fa74355f" Nov 02 17:56:54.658139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.658119 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" event={"ID":"3fc14d78-414c-48da-8d5e-0828ea8cc5c1","Type":"ContainerStarted","Data":"19e6bdfe539950d9ec835083be12fe9f43d7749e2636d10f07026528a6bbb6d3"} Nov 02 17:56:54.764039 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.763987 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:54.999940 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:54.999909 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:55.092423 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.092388 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-xks-ca\") pod \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " Nov 02 17:56:55.092594 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.092444 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1183c12f-ff3c-4f0c-adf0-a0883c99e839-kube-api-access-4t484\" (UniqueName: \"kubernetes.io/projected/1183c12f-ff3c-4f0c-adf0-a0883c99e839-kube-api-access-4t484\") pod \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " Nov 02 17:56:55.092663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.092651 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-openshift-ca\") pod \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\" (UID: \"1183c12f-ff3c-4f0c-adf0-a0883c99e839\") " Nov 02 17:56:55.092972 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.092941 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-openshift-ca" pod "1183c12f-ff3c-4f0c-adf0-a0883c99e839" (UID: "1183c12f-ff3c-4f0c-adf0-a0883c99e839"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:55.093280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.093260 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:55.094620 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.094596 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-xks-ca" pod "1183c12f-ff3c-4f0c-adf0-a0883c99e839" (UID: "1183c12f-ff3c-4f0c-adf0-a0883c99e839"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:55.094718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.094617 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1183c12f-ff3c-4f0c-adf0-a0883c99e839-kube-api-access-4t484" pod "1183c12f-ff3c-4f0c-adf0-a0883c99e839" (UID: "1183c12f-ff3c-4f0c-adf0-a0883c99e839"). InnerVolumeSpecName "kube-api-access-4t484". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:55.194647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.194615 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1183c12f-ff3c-4f0c-adf0-a0883c99e839-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:55.194647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.194643 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t484\" (UniqueName: \"kubernetes.io/projected/1183c12f-ff3c-4f0c-adf0-a0883c99e839-kube-api-access-4t484\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:55.570631 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.570598 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:55.757735 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.757702 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" event={"ID":"1183c12f-ff3c-4f0c-adf0-a0883c99e839","Type":"ContainerDied","Data":"870b2413a4233fb349c7d243f7d4b00bb4ead12ccd513548f9f84ccbcd3a1a3a"} Nov 02 17:56:55.758191 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.758175 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="870b2413a4233fb349c7d243f7d4b00bb4ead12ccd513548f9f84ccbcd3a1a3a" Nov 02 17:56:55.758335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.758325 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77" Nov 02 17:56:55.761902 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.761869 4310 generic.go:356] "Generic (PLEG): container finished" podID="3fc14d78-414c-48da-8d5e-0828ea8cc5c1" containerID="1b6d3cc6bba1ce9445fc2272044c70e5db0835bfb38285f46010e35df003f26a" exitCode=0 Nov 02 17:56:55.762067 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.762046 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" event={"ID":"3fc14d78-414c-48da-8d5e-0828ea8cc5c1","Type":"ContainerDied","Data":"1b6d3cc6bba1ce9445fc2272044c70e5db0835bfb38285f46010e35df003f26a"} Nov 02 17:56:55.774064 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.774028 4310 generic.go:356] "Generic (PLEG): container finished" podID="688fa1f2-1628-404a-9a23-39640c88484c" containerID="0cc318c9075f213f9eef1f2d5d6282ec3cc6168637b95fb443f7c6c03d3eb9b2" exitCode=0 Nov 02 17:56:55.774318 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:55.774295 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" event={"ID":"688fa1f2-1628-404a-9a23-39640c88484c","Type":"ContainerDied","Data":"0cc318c9075f213f9eef1f2d5d6282ec3cc6168637b95fb443f7c6c03d3eb9b2"} Nov 02 17:56:56.371603 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.371540 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" containerID="cri-o://93325fa53b7512224a896f193fe2fe5191d5e1bfd2b696b8637d63a9316da83a" gracePeriod=45 Nov 02 17:56:56.389996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.389953 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" containerID="cri-o://7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253" gracePeriod=45 Nov 02 17:56:56.415142 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.415110 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:56.791726 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.791694 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:56:56.797726 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.797695 4310 generic.go:356] "Generic (PLEG): container finished" podID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerID="7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253" exitCode=0 Nov 02 17:56:56.797862 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.797730 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" event={"ID":"3a96b395-eab3-4efb-8b1a-dda4c909f13d","Type":"ContainerDied","Data":"7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253"} Nov 02 17:56:56.797862 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.797782 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" event={"ID":"3a96b395-eab3-4efb-8b1a-dda4c909f13d","Type":"ContainerDied","Data":"ab8b854edc991188fc4cecdc0a3c1330e3f2457464d8d7ad4734cd956e002395"} Nov 02 17:56:56.797862 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.797804 4310 scope.go:122] "RemoveContainer" containerID="7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253" Nov 02 17:56:56.798036 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.798015 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" Nov 02 17:56:56.800071 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.800043 4310 generic.go:356] "Generic (PLEG): container finished" podID="89a87b59-04df-4013-8602-e50f5ce2842a" containerID="93325fa53b7512224a896f193fe2fe5191d5e1bfd2b696b8637d63a9316da83a" exitCode=0 Nov 02 17:56:56.800204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.800138 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" event={"ID":"89a87b59-04df-4013-8602-e50f5ce2842a","Type":"ContainerDied","Data":"93325fa53b7512224a896f193fe2fe5191d5e1bfd2b696b8637d63a9316da83a"} Nov 02 17:56:56.853925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.853857 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-home\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-home\") pod \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " Nov 02 17:56:56.854884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.854174 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-model-cache\") pod \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " Nov 02 17:56:56.854884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.854320 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tls-certs\") pod \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " Nov 02 17:56:56.854884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.854497 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-dshm\") pod \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " Nov 02 17:56:56.854884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.854529 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a96b395-eab3-4efb-8b1a-dda4c909f13d-kube-api-access-6x899\" (UniqueName: \"kubernetes.io/projected/3a96b395-eab3-4efb-8b1a-dda4c909f13d-kube-api-access-6x899\") pod \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " Nov 02 17:56:56.854884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.854578 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tmp-dir\") pod \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\" (UID: \"3a96b395-eab3-4efb-8b1a-dda4c909f13d\") " Nov 02 17:56:56.856246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.856006 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tmp-dir" pod "3a96b395-eab3-4efb-8b1a-dda4c909f13d" (UID: "3a96b395-eab3-4efb-8b1a-dda4c909f13d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.856577 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.856382 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-home" pod "3a96b395-eab3-4efb-8b1a-dda4c909f13d" (UID: "3a96b395-eab3-4efb-8b1a-dda4c909f13d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.857007 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.856707 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-model-cache" pod "3a96b395-eab3-4efb-8b1a-dda4c909f13d" (UID: "3a96b395-eab3-4efb-8b1a-dda4c909f13d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.862699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.861411 4310 scope.go:122] "RemoveContainer" containerID="7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253" Nov 02 17:56:56.863725 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:56:56.863567 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253\": container with ID starting with 7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253 not found: ID does not exist" containerID="7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253" Nov 02 17:56:56.863725 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.863613 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253"} err="failed to get container status \"7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253\": rpc error: code = NotFound desc = could not find container \"7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253\": container with ID starting with 7422b23009ef9603376323011a93a2ba879714839a86d2f34572a103d72be253 not found: ID does not exist" Nov 02 17:56:56.866365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.866341 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a96b395-eab3-4efb-8b1a-dda4c909f13d-kube-api-access-6x899" pod "3a96b395-eab3-4efb-8b1a-dda4c909f13d" (UID: "3a96b395-eab3-4efb-8b1a-dda4c909f13d"). InnerVolumeSpecName "kube-api-access-6x899". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:56.869000 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.868972 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tls-certs" pod "3a96b395-eab3-4efb-8b1a-dda4c909f13d" (UID: "3a96b395-eab3-4efb-8b1a-dda4c909f13d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:56.870253 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.870229 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:56:56.875009 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.873521 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-dshm" pod "3a96b395-eab3-4efb-8b1a-dda4c909f13d" (UID: "3a96b395-eab3-4efb-8b1a-dda4c909f13d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.957067 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.957020 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-tmp-dir\") pod \"89a87b59-04df-4013-8602-e50f5ce2842a\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " Nov 02 17:56:56.957244 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.957204 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89a87b59-04df-4013-8602-e50f5ce2842a-tls-certs\" (UniqueName: \"kubernetes.io/secret/89a87b59-04df-4013-8602-e50f5ce2842a-tls-certs\") pod \"89a87b59-04df-4013-8602-e50f5ce2842a\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " Nov 02 17:56:56.957408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.957380 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-tmp-dir" pod "89a87b59-04df-4013-8602-e50f5ce2842a" (UID: "89a87b59-04df-4013-8602-e50f5ce2842a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.957446 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.957405 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-home\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-home\") pod \"89a87b59-04df-4013-8602-e50f5ce2842a\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " Nov 02 17:56:56.957489 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.957458 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a87b59-04df-4013-8602-e50f5ce2842a-kube-api-access-jtnwc\" (UniqueName: \"kubernetes.io/projected/89a87b59-04df-4013-8602-e50f5ce2842a-kube-api-access-jtnwc\") pod \"89a87b59-04df-4013-8602-e50f5ce2842a\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " Nov 02 17:56:56.957528 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.957514 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-dshm\") pod \"89a87b59-04df-4013-8602-e50f5ce2842a\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " Nov 02 17:56:56.957573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.957539 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-model-cache\") pod \"89a87b59-04df-4013-8602-e50f5ce2842a\" (UID: \"89a87b59-04df-4013-8602-e50f5ce2842a\") " Nov 02 17:56:56.958213 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.958185 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-home" pod "89a87b59-04df-4013-8602-e50f5ce2842a" (UID: "89a87b59-04df-4013-8602-e50f5ce2842a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.958440 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.958417 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-model-cache" pod "89a87b59-04df-4013-8602-e50f5ce2842a" (UID: "89a87b59-04df-4013-8602-e50f5ce2842a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:56.958619 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.958596 4310 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tls-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.958887 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.958873 4310 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-dshm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.959008 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.958996 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x899\" (UniqueName: \"kubernetes.io/projected/3a96b395-eab3-4efb-8b1a-dda4c909f13d-kube-api-access-6x899\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.959124 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.959112 4310 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-tmp-dir\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.959204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.959195 4310 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-home\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.959289 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.959279 4310 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-home\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.959365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.959355 4310 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a96b395-eab3-4efb-8b1a-dda4c909f13d-model-cache\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.959448 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.959438 4310 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-tmp-dir\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:56.962545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.960686 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a87b59-04df-4013-8602-e50f5ce2842a-tls-certs" pod "89a87b59-04df-4013-8602-e50f5ce2842a" (UID: "89a87b59-04df-4013-8602-e50f5ce2842a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:56.962545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.961270 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a87b59-04df-4013-8602-e50f5ce2842a-kube-api-access-jtnwc" pod "89a87b59-04df-4013-8602-e50f5ce2842a" (UID: "89a87b59-04df-4013-8602-e50f5ce2842a"). InnerVolumeSpecName "kube-api-access-jtnwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:56.966175 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:56.966145 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-dshm" pod "89a87b59-04df-4013-8602-e50f5ce2842a" (UID: "89a87b59-04df-4013-8602-e50f5ce2842a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:57.061310 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.061255 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtnwc\" (UniqueName: \"kubernetes.io/projected/89a87b59-04df-4013-8602-e50f5ce2842a-kube-api-access-jtnwc\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:57.061310 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.061292 4310 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-dshm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:57.061310 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.061309 4310 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a87b59-04df-4013-8602-e50f5ce2842a-model-cache\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:57.061574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.061327 4310 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a87b59-04df-4013-8602-e50f5ce2842a-tls-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:57.076190 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.075606 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:57.113572 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.113550 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:57.133825 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.133784 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9"] Nov 02 17:56:57.145375 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.145297 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9"] Nov 02 17:56:57.161891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.161858 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-xks-ca\") pod \"688fa1f2-1628-404a-9a23-39640c88484c\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " Nov 02 17:56:57.162048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.161904 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-kube-api-access-w9sks\" (UniqueName: \"kubernetes.io/projected/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-kube-api-access-w9sks\") pod \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " Nov 02 17:56:57.162048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.161945 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-openshift-ca\") pod \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " Nov 02 17:56:57.162048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.161961 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-openshift-ca\") pod \"688fa1f2-1628-404a-9a23-39640c88484c\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " Nov 02 17:56:57.162048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.161996 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/688fa1f2-1628-404a-9a23-39640c88484c-kube-api-access-7kl7r\" (UniqueName: \"kubernetes.io/projected/688fa1f2-1628-404a-9a23-39640c88484c-kube-api-access-7kl7r\") pod \"688fa1f2-1628-404a-9a23-39640c88484c\" (UID: \"688fa1f2-1628-404a-9a23-39640c88484c\") " Nov 02 17:56:57.162318 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.162277 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-xks-ca\") pod \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\" (UID: \"3fc14d78-414c-48da-8d5e-0828ea8cc5c1\") " Nov 02 17:56:57.163002 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.162960 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-openshift-ca" pod "3fc14d78-414c-48da-8d5e-0828ea8cc5c1" (UID: "3fc14d78-414c-48da-8d5e-0828ea8cc5c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:57.163192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.163164 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:57.163652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.163627 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-openshift-ca" pod "688fa1f2-1628-404a-9a23-39640c88484c" (UID: "688fa1f2-1628-404a-9a23-39640c88484c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:57.165540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.165502 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-kube-api-access-w9sks" pod "3fc14d78-414c-48da-8d5e-0828ea8cc5c1" (UID: "3fc14d78-414c-48da-8d5e-0828ea8cc5c1"). InnerVolumeSpecName "kube-api-access-w9sks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:57.165781 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.165746 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688fa1f2-1628-404a-9a23-39640c88484c-kube-api-access-7kl7r" pod "688fa1f2-1628-404a-9a23-39640c88484c" (UID: "688fa1f2-1628-404a-9a23-39640c88484c"). InnerVolumeSpecName "kube-api-access-7kl7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:57.167431 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.167411 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-xks-ca" pod "688fa1f2-1628-404a-9a23-39640c88484c" (UID: "688fa1f2-1628-404a-9a23-39640c88484c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:57.167703 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.167677 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-xks-ca" pod "3fc14d78-414c-48da-8d5e-0828ea8cc5c1" (UID: "3fc14d78-414c-48da-8d5e-0828ea8cc5c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:57.264386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.264356 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:57.264386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.264384 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9sks\" (UniqueName: \"kubernetes.io/projected/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-kube-api-access-w9sks\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:57.264386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.264394 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688fa1f2-1628-404a-9a23-39640c88484c-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:57.264621 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.264406 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kl7r\" (UniqueName: \"kubernetes.io/projected/688fa1f2-1628-404a-9a23-39640c88484c-kube-api-access-7kl7r\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:57.264621 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.264419 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc14d78-414c-48da-8d5e-0828ea8cc5c1-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:57.357390 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.357349 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:57.650149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.650112 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" containerID="cri-o://7283f8e404ba2ee70ae19f9d422844c8b180475ccacaa2094d0ddbca17ba086c" gracePeriod=45 Nov 02 17:56:57.661698 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.661628 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" containerID="cri-o://2f137990694d17a26e14c43eab84468233d19306643fcb84a0ea46ac3d6a0ecd" gracePeriod=45 Nov 02 17:56:57.744188 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.744146 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:56:57.783201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.783163 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f37d16/simulator-w0-f37d16-kserve-5c87ffdd66-c7rx9" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:56:57.815845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.815812 4310 generic.go:356] "Generic (PLEG): container finished" podID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerID="7283f8e404ba2ee70ae19f9d422844c8b180475ccacaa2094d0ddbca17ba086c" exitCode=0 Nov 02 17:56:57.816228 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.815898 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" event={"ID":"1bdb0ab7-10bd-4211-b99a-0cc15697684e","Type":"ContainerDied","Data":"7283f8e404ba2ee70ae19f9d422844c8b180475ccacaa2094d0ddbca17ba086c"} Nov 02 17:56:57.817586 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.817559 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" Nov 02 17:56:57.817744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.817588 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq" event={"ID":"688fa1f2-1628-404a-9a23-39640c88484c","Type":"ContainerDied","Data":"16f8f945638873cb181c357c5c03cdb92e3bd5ccc9e2c96da3b2ae79308d37e0"} Nov 02 17:56:57.817744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.817636 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16f8f945638873cb181c357c5c03cdb92e3bd5ccc9e2c96da3b2ae79308d37e0" Nov 02 17:56:57.821132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.820253 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" event={"ID":"89a87b59-04df-4013-8602-e50f5ce2842a","Type":"ContainerDied","Data":"db79a730694070d7f6108c2eaca497a4192b89bbb0b87cf80c1adee2d727f2c8"} Nov 02 17:56:57.821132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.820299 4310 scope.go:122] "RemoveContainer" containerID="93325fa53b7512224a896f193fe2fe5191d5e1bfd2b696b8637d63a9316da83a" Nov 02 17:56:57.821132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.820407 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf" Nov 02 17:56:57.823902 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.823875 4310 generic.go:356] "Generic (PLEG): container finished" podID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerID="2f137990694d17a26e14c43eab84468233d19306643fcb84a0ea46ac3d6a0ecd" exitCode=0 Nov 02 17:56:57.824153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.824134 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" event={"ID":"80e9e16c-4460-4acf-8c44-bae8d73e09dd","Type":"ContainerDied","Data":"2f137990694d17a26e14c43eab84468233d19306643fcb84a0ea46ac3d6a0ecd"} Nov 02 17:56:57.826004 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.825979 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" event={"ID":"3fc14d78-414c-48da-8d5e-0828ea8cc5c1","Type":"ContainerDied","Data":"19e6bdfe539950d9ec835083be12fe9f43d7749e2636d10f07026528a6bbb6d3"} Nov 02 17:56:57.826145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.826024 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19e6bdfe539950d9ec835083be12fe9f43d7749e2636d10f07026528a6bbb6d3" Nov 02 17:56:57.826145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.826101 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6" Nov 02 17:56:57.890753 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.890219 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf"] Nov 02 17:56:57.915171 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:57.908540 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f37d16/premium-w0-f37d16-kserve-6f6b9bb58c-bnqmf"] Nov 02 17:56:58.022679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.022575 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:56:58.044496 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.027181 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:56:58.044496 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.044297 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b"] Nov 02 17:56:58.054335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.053486 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ef14ca-drt2b"] Nov 02 17:56:58.078203 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.078168 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-model-cache\") pod \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " Nov 02 17:56:58.078459 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.078224 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-dshm\") pod \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " Nov 02 17:56:58.078459 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.078434 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tmp-dir\") pod \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " Nov 02 17:56:58.078622 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.078604 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tmp-dir\") pod \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " Nov 02 17:56:58.078784 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.078691 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bdb0ab7-10bd-4211-b99a-0cc15697684e-kube-api-access-vhdc5\" (UniqueName: \"kubernetes.io/projected/1bdb0ab7-10bd-4211-b99a-0cc15697684e-kube-api-access-vhdc5\") pod \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " Nov 02 17:56:58.078865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.078818 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-dshm\") pod \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " Nov 02 17:56:58.078865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.078849 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-home\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-home\") pod \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " Nov 02 17:56:58.078972 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.078881 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-model-cache" pod "80e9e16c-4460-4acf-8c44-bae8d73e09dd" (UID: "80e9e16c-4460-4acf-8c44-bae8d73e09dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.078972 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.078903 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-model-cache\") pod \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " Nov 02 17:56:58.079144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.078977 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tls-certs\") pod \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " Nov 02 17:56:58.079144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.079013 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e9e16c-4460-4acf-8c44-bae8d73e09dd-kube-api-access-8tvdt\" (UniqueName: \"kubernetes.io/projected/80e9e16c-4460-4acf-8c44-bae8d73e09dd-kube-api-access-8tvdt\") pod \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " Nov 02 17:56:58.079267 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.079190 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-home\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-home\") pod \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\" (UID: \"80e9e16c-4460-4acf-8c44-bae8d73e09dd\") " Nov 02 17:56:58.079330 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.079285 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tls-certs\") pod \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\" (UID: \"1bdb0ab7-10bd-4211-b99a-0cc15697684e\") " Nov 02 17:56:58.079573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.079396 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-model-cache" pod "1bdb0ab7-10bd-4211-b99a-0cc15697684e" (UID: "1bdb0ab7-10bd-4211-b99a-0cc15697684e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.079573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.079428 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tmp-dir" pod "80e9e16c-4460-4acf-8c44-bae8d73e09dd" (UID: "80e9e16c-4460-4acf-8c44-bae8d73e09dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.079864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.079600 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-home" pod "1bdb0ab7-10bd-4211-b99a-0cc15697684e" (UID: "1bdb0ab7-10bd-4211-b99a-0cc15697684e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.079950 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.079900 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tmp-dir" pod "1bdb0ab7-10bd-4211-b99a-0cc15697684e" (UID: "1bdb0ab7-10bd-4211-b99a-0cc15697684e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.080333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.080091 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-home" pod "80e9e16c-4460-4acf-8c44-bae8d73e09dd" (UID: "80e9e16c-4460-4acf-8c44-bae8d73e09dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.080333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.080290 4310 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-model-cache\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.080333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.080314 4310 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tmp-dir\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.080333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.080328 4310 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tmp-dir\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.080574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.080340 4310 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-home\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.080574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.080352 4310 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-model-cache\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.088554 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.088523 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tls-certs" pod "1bdb0ab7-10bd-4211-b99a-0cc15697684e" (UID: "1bdb0ab7-10bd-4211-b99a-0cc15697684e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:58.088554 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.088543 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tls-certs" pod "80e9e16c-4460-4acf-8c44-bae8d73e09dd" (UID: "80e9e16c-4460-4acf-8c44-bae8d73e09dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:58.088744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.088706 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e9e16c-4460-4acf-8c44-bae8d73e09dd-kube-api-access-8tvdt" pod "80e9e16c-4460-4acf-8c44-bae8d73e09dd" (UID: "80e9e16c-4460-4acf-8c44-bae8d73e09dd"). InnerVolumeSpecName "kube-api-access-8tvdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:58.088823 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.088797 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bdb0ab7-10bd-4211-b99a-0cc15697684e-kube-api-access-vhdc5" pod "1bdb0ab7-10bd-4211-b99a-0cc15697684e" (UID: "1bdb0ab7-10bd-4211-b99a-0cc15697684e"). InnerVolumeSpecName "kube-api-access-vhdc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:58.088945 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.088908 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-dshm" pod "80e9e16c-4460-4acf-8c44-bae8d73e09dd" (UID: "80e9e16c-4460-4acf-8c44-bae8d73e09dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.090673 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.090647 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-dshm" pod "1bdb0ab7-10bd-4211-b99a-0cc15697684e" (UID: "1bdb0ab7-10bd-4211-b99a-0cc15697684e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:58.112923 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.112894 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:58.182028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.181949 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhdc5\" (UniqueName: \"kubernetes.io/projected/1bdb0ab7-10bd-4211-b99a-0cc15697684e-kube-api-access-vhdc5\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.182028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.181974 4310 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdb0ab7-10bd-4211-b99a-0cc15697684e-dshm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.182028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.181983 4310 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e9e16c-4460-4acf-8c44-bae8d73e09dd-tls-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.182028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.181992 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tvdt\" (UniqueName: \"kubernetes.io/projected/80e9e16c-4460-4acf-8c44-bae8d73e09dd-kube-api-access-8tvdt\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.182028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.182000 4310 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-home\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.182028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.182008 4310 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdb0ab7-10bd-4211-b99a-0cc15697684e-tls-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.182028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.182016 4310 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e9e16c-4460-4acf-8c44-bae8d73e09dd-dshm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:56:58.676833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.676793 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" path="/var/lib/kubelet/pods/3a96b395-eab3-4efb-8b1a-dda4c909f13d/volumes" Nov 02 17:56:58.677355 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.677336 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e92a80-24f9-4a24-b6b2-5e53b21879b2" path="/var/lib/kubelet/pods/81e92a80-24f9-4a24-b6b2-5e53b21879b2/volumes" Nov 02 17:56:58.677755 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.677742 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" path="/var/lib/kubelet/pods/89a87b59-04df-4013-8602-e50f5ce2842a/volumes" Nov 02 17:56:58.836595 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.836560 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" Nov 02 17:56:58.836595 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.836580 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" event={"ID":"80e9e16c-4460-4acf-8c44-bae8d73e09dd","Type":"ContainerDied","Data":"e8bf3965e3f78b529b353e76e4ecf8841b9ce84657d3daa18d4b6a8084e51f90"} Nov 02 17:56:58.837164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.836642 4310 scope.go:122] "RemoveContainer" containerID="2f137990694d17a26e14c43eab84468233d19306643fcb84a0ea46ac3d6a0ecd" Nov 02 17:56:58.838513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.838492 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" Nov 02 17:56:58.838602 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.838524 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m" event={"ID":"1bdb0ab7-10bd-4211-b99a-0cc15697684e","Type":"ContainerDied","Data":"8e97838e7e25fd7f8bd496adca57b8f40937e97bc0eb6283e085aaa5db01ddea"} Nov 02 17:56:58.855612 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.855591 4310 scope.go:122] "RemoveContainer" containerID="7283f8e404ba2ee70ae19f9d422844c8b180475ccacaa2094d0ddbca17ba086c" Nov 02 17:56:58.867611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.867570 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:56:58.871086 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.871055 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4"] Nov 02 17:56:58.877002 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.876975 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4"] Nov 02 17:56:58.887562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.887540 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m"] Nov 02 17:56:58.893172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.893147 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ea3867/distinct-2-w5-ea3867-kserve-894d86c7-zms5m"] Nov 02 17:56:58.989139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:58.984401 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ea3867/distinct-w5-ea3867-kserve-65bf875d78-fwxs4" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 02 17:56:59.580261 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:56:59.580226 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:00.411144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:00.411115 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:00.677190 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:00.677101 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" path="/var/lib/kubelet/pods/1bdb0ab7-10bd-4211-b99a-0cc15697684e/volumes" Nov 02 17:57:00.677817 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:00.677793 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" path="/var/lib/kubelet/pods/80e9e16c-4460-4acf-8c44-bae8d73e09dd/volumes" Nov 02 17:57:01.116478 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:01.116438 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:01.260099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:01.253374 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" podUID="b453de3f-2c9c-4705-ab30-b3eae093a0b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:57:01.269150 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:01.267190 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" podUID="2469cdd9-c68f-477b-9cb8-85aadba1bacd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:57:01.882811 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:01.882773 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:02.222301 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.222260 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" podUID="1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:57:02.670657 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.670555 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:02.841173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.841126 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq"] Nov 02 17:57:02.842187 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.842161 4310 cpu_manager.go:414] "containerMap: removing container" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" Nov 02 17:57:02.842360 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.842344 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" Nov 02 17:57:02.842473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.842461 4310 cpu_manager.go:414] "containerMap: removing container" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" Nov 02 17:57:02.842553 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.842543 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" Nov 02 17:57:02.842646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.842636 4310 cpu_manager.go:414] "containerMap: removing container" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" Nov 02 17:57:02.842731 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.842719 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" Nov 02 17:57:02.842845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.842834 4310 cpu_manager.go:414] "containerMap: removing container" podUID="3fc14d78-414c-48da-8d5e-0828ea8cc5c1" containerName="revoke-keys" Nov 02 17:57:02.842920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.842909 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc14d78-414c-48da-8d5e-0828ea8cc5c1" containerName="revoke-keys" Nov 02 17:57:02.843002 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.842993 4310 cpu_manager.go:414] "containerMap: removing container" podUID="b6f80910-45f6-436c-b553-a41ebe2808ea" containerName="revoke-keys" Nov 02 17:57:02.843090 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.843065 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f80910-45f6-436c-b553-a41ebe2808ea" containerName="revoke-keys" Nov 02 17:57:02.843189 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.843174 4310 cpu_manager.go:414] "containerMap: removing container" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" Nov 02 17:57:02.843293 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.843282 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" Nov 02 17:57:02.843381 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.843371 4310 cpu_manager.go:414] "containerMap: removing container" podUID="1183c12f-ff3c-4f0c-adf0-a0883c99e839" containerName="revoke-keys" Nov 02 17:57:02.843448 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.843439 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1183c12f-ff3c-4f0c-adf0-a0883c99e839" containerName="revoke-keys" Nov 02 17:57:02.843537 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.843529 4310 cpu_manager.go:414] "containerMap: removing container" podUID="688fa1f2-1628-404a-9a23-39640c88484c" containerName="revoke-keys" Nov 02 17:57:02.843611 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.843602 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="688fa1f2-1628-404a-9a23-39640c88484c" containerName="revoke-keys" Nov 02 17:57:02.843885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.843870 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a87b59-04df-4013-8602-e50f5ce2842a" containerName="main" Nov 02 17:57:02.843985 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.843975 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e9e16c-4460-4acf-8c44-bae8d73e09dd" containerName="main" Nov 02 17:57:02.844052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.844046 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc14d78-414c-48da-8d5e-0828ea8cc5c1" containerName="revoke-keys" Nov 02 17:57:02.844139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.844128 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f80910-45f6-436c-b553-a41ebe2808ea" containerName="revoke-keys" Nov 02 17:57:02.844221 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.844210 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a96b395-eab3-4efb-8b1a-dda4c909f13d" containerName="main" Nov 02 17:57:02.844364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.844303 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="1183c12f-ff3c-4f0c-adf0-a0883c99e839" containerName="revoke-keys" Nov 02 17:57:02.844508 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.844491 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="688fa1f2-1628-404a-9a23-39640c88484c" containerName="revoke-keys" Nov 02 17:57:02.844634 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.844621 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bdb0ab7-10bd-4211-b99a-0cc15697684e" containerName="main" Nov 02 17:57:02.856796 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.856770 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:02.859647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.859619 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq"] Nov 02 17:57:02.867103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.863843 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:57:02.867103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.864709 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:57:02.886040 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.886008 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46288\" (UniqueName: \"kubernetes.io/projected/6e328e69-eeb5-4258-81c3-7a56397c0563-kube-api-access-46288\") pod \"maas-api-revoke-sub-e2fa5f65-f2psq\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:02.886405 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.886063 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fa5f65-f2psq\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:02.886405 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.886354 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fa5f65-f2psq\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:02.987653 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.987621 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46288\" (UniqueName: \"kubernetes.io/projected/6e328e69-eeb5-4258-81c3-7a56397c0563-kube-api-access-46288\") pod \"maas-api-revoke-sub-e2fa5f65-f2psq\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:02.987815 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.987672 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fa5f65-f2psq\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:02.987815 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.987796 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fa5f65-f2psq\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:02.988303 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.988283 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fa5f65-f2psq\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:02.990197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.990170 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fa5f65-f2psq\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:02.996835 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:02.996815 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46288\" (UniqueName: \"kubernetes.io/projected/6e328e69-eeb5-4258-81c3-7a56397c0563-kube-api-access-46288\") pod \"maas-api-revoke-sub-e2fa5f65-f2psq\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:03.176427 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:03.176385 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:03.483479 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:03.483449 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:03.487177 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:57:03.487145 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e328e69_eeb5_4258_81c3_7a56397c0563.slice/crio-a2982d97b99497eaef5fb68eafbc1999892de0454efa1a2bd49503ea26635e1d WatchSource:0}: Error finding container a2982d97b99497eaef5fb68eafbc1999892de0454efa1a2bd49503ea26635e1d: Status 404 returned error can't find the container with id a2982d97b99497eaef5fb68eafbc1999892de0454efa1a2bd49503ea26635e1d Nov 02 17:57:03.489411 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:03.489382 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq"] Nov 02 17:57:03.577100 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:57:03.575669 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:57:03.895243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:03.895198 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" event={"ID":"6e328e69-eeb5-4258-81c3-7a56397c0563","Type":"ContainerStarted","Data":"509af34add89ddc522002157c0349cd3cc11da75e4c38561d7a56edab9861328"} Nov 02 17:57:03.895624 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:03.895304 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" event={"ID":"6e328e69-eeb5-4258-81c3-7a56397c0563","Type":"ContainerStarted","Data":"a2982d97b99497eaef5fb68eafbc1999892de0454efa1a2bd49503ea26635e1d"} Nov 02 17:57:03.916509 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:03.916462 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" podStartSLOduration=1.916448396 podStartE2EDuration="1.916448396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:03.912039152 +0000 UTC m=+1021.355121832" watchObservedRunningTime="2026-11-02 17:57:03.916448396 +0000 UTC m=+1021.359531031" Nov 02 17:57:04.205909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:04.205830 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:04.867449 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:04.867422 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:04.913580 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:04.913539 4310 generic.go:356] "Generic (PLEG): container finished" podID="6e328e69-eeb5-4258-81c3-7a56397c0563" containerID="509af34add89ddc522002157c0349cd3cc11da75e4c38561d7a56edab9861328" exitCode=0 Nov 02 17:57:04.914045 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:04.913645 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" event={"ID":"6e328e69-eeb5-4258-81c3-7a56397c0563","Type":"ContainerDied","Data":"509af34add89ddc522002157c0349cd3cc11da75e4c38561d7a56edab9861328"} Nov 02 17:57:05.558995 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:05.558952 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:06.088811 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.088787 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:06.171484 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.171449 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e328e69-eeb5-4258-81c3-7a56397c0563-kube-api-access-46288\" (UniqueName: \"kubernetes.io/projected/6e328e69-eeb5-4258-81c3-7a56397c0563-kube-api-access-46288\") pod \"6e328e69-eeb5-4258-81c3-7a56397c0563\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " Nov 02 17:57:06.171668 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.171565 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-xks-ca\") pod \"6e328e69-eeb5-4258-81c3-7a56397c0563\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " Nov 02 17:57:06.171753 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.171726 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-openshift-ca\") pod \"6e328e69-eeb5-4258-81c3-7a56397c0563\" (UID: \"6e328e69-eeb5-4258-81c3-7a56397c0563\") " Nov 02 17:57:06.172538 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.172491 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-openshift-ca" pod "6e328e69-eeb5-4258-81c3-7a56397c0563" (UID: "6e328e69-eeb5-4258-81c3-7a56397c0563"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:06.174243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.174182 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e328e69-eeb5-4258-81c3-7a56397c0563-kube-api-access-46288" pod "6e328e69-eeb5-4258-81c3-7a56397c0563" (UID: "6e328e69-eeb5-4258-81c3-7a56397c0563"). InnerVolumeSpecName "kube-api-access-46288". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:06.174344 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.174237 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-xks-ca" pod "6e328e69-eeb5-4258-81c3-7a56397c0563" (UID: "6e328e69-eeb5-4258-81c3-7a56397c0563"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:06.274252 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.274212 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46288\" (UniqueName: \"kubernetes.io/projected/6e328e69-eeb5-4258-81c3-7a56397c0563-kube-api-access-46288\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:06.274252 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.274249 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:06.274252 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.274260 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e328e69-eeb5-4258-81c3-7a56397c0563-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:06.361976 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.361941 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:06.933518 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.933473 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" Nov 02 17:57:06.933668 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.933482 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq" event={"ID":"6e328e69-eeb5-4258-81c3-7a56397c0563","Type":"ContainerDied","Data":"a2982d97b99497eaef5fb68eafbc1999892de0454efa1a2bd49503ea26635e1d"} Nov 02 17:57:06.933668 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:06.933630 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2982d97b99497eaef5fb68eafbc1999892de0454efa1a2bd49503ea26635e1d" Nov 02 17:57:07.008844 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:07.008800 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:07.711791 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:07.711753 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:08.050878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:08.050806 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp"] Nov 02 17:57:08.060563 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:08.060537 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz"] Nov 02 17:57:08.065873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:08.065850 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5c5a11-v9zdp"] Nov 02 17:57:08.070622 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:08.070598 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2438355-2twrz"] Nov 02 17:57:08.601841 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:08.601804 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:08.772781 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:08.772232 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318ca266-1cfd-451a-9e7c-b2145e9c512c" path="/var/lib/kubelet/pods/318ca266-1cfd-451a-9e7c-b2145e9c512c/volumes" Nov 02 17:57:08.773258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:08.772957 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc882aa-7cbf-4d3a-85eb-5d6459f787ab" path="/var/lib/kubelet/pods/dfc882aa-7cbf-4d3a-85eb-5d6459f787ab/volumes" Nov 02 17:57:09.486837 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:09.486806 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:09.667100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:09.665044 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:09.994716 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:09.994683 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:10.003840 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:10.003812 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p"] Nov 02 17:57:10.013727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:10.013677 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f37d16-7dm7p"] Nov 02 17:57:10.145106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:10.141635 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:10.498811 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:10.498779 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:10.678767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:10.677836 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163da1e2-9ee5-49d0-86ea-4069654a73f4" path="/var/lib/kubelet/pods/163da1e2-9ee5-49d0-86ea-4069654a73f4/volumes" Nov 02 17:57:11.214556 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:11.214520 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:11.216600 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:11.216563 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" podUID="2469cdd9-c68f-477b-9cb8-85aadba1bacd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:57:11.216856 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:11.216828 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" podUID="b453de3f-2c9c-4705-ab30-b3eae093a0b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:57:12.116006 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:12.115974 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:12.225306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:12.225241 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" podUID="1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:57:12.930804 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:12.930549 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:12.994287 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:12.994258 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:57:13.012207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.012180 4310 generic.go:356] "Generic (PLEG): container finished" podID="2469cdd9-c68f-477b-9cb8-85aadba1bacd" containerID="b57773de42695ac73c95e591f83f2ea173a22f3600d650794563c869c45f92ae" exitCode=0 Nov 02 17:57:13.012317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.012251 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" event={"ID":"2469cdd9-c68f-477b-9cb8-85aadba1bacd","Type":"ContainerDied","Data":"b57773de42695ac73c95e591f83f2ea173a22f3600d650794563c869c45f92ae"} Nov 02 17:57:13.012317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.012296 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" event={"ID":"2469cdd9-c68f-477b-9cb8-85aadba1bacd","Type":"ContainerDied","Data":"cc35c366869f6bb171efe41ffbac1a64e17d9d1d950f0ed5ef0017f4e754f596"} Nov 02 17:57:13.012317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.012313 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc35c366869f6bb171efe41ffbac1a64e17d9d1d950f0ed5ef0017f4e754f596" Nov 02 17:57:13.013940 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.013920 4310 generic.go:356] "Generic (PLEG): container finished" podID="b453de3f-2c9c-4705-ab30-b3eae093a0b0" containerID="b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a" exitCode=0 Nov 02 17:57:13.014046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.013961 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" Nov 02 17:57:13.014046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.014005 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" event={"ID":"b453de3f-2c9c-4705-ab30-b3eae093a0b0","Type":"ContainerDied","Data":"b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a"} Nov 02 17:57:13.014148 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.014059 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn" event={"ID":"b453de3f-2c9c-4705-ab30-b3eae093a0b0","Type":"ContainerDied","Data":"0b7cccc334807c2a96d493f3442e27caa6abd2509f51286ca5df5e8ddd2f1ad1"} Nov 02 17:57:13.014148 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.014094 4310 scope.go:122] "RemoveContainer" containerID="b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a" Nov 02 17:57:13.024639 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.024622 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:57:13.026679 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.026656 4310 scope.go:122] "RemoveContainer" containerID="b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a" Nov 02 17:57:13.026944 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:57:13.026924 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a\": container with ID starting with b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a not found: ID does not exist" containerID="b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a" Nov 02 17:57:13.027020 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.026956 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a"} err="failed to get container status \"b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a\": rpc error: code = NotFound desc = could not find container \"b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a\": container with ID starting with b2d650886ee5f03e156000bc88759afce8b4f46412aeb787d12090dc81bfa02a not found: ID does not exist" Nov 02 17:57:13.029470 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.029445 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b453de3f-2c9c-4705-ab30-b3eae093a0b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b453de3f-2c9c-4705-ab30-b3eae093a0b0-plugins-config-volume\") pod \"b453de3f-2c9c-4705-ab30-b3eae093a0b0\" (UID: \"b453de3f-2c9c-4705-ab30-b3eae093a0b0\") " Nov 02 17:57:13.029682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.029665 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b453de3f-2c9c-4705-ab30-b3eae093a0b0-kube-api-access-fbhm2\" (UniqueName: \"kubernetes.io/projected/b453de3f-2c9c-4705-ab30-b3eae093a0b0-kube-api-access-fbhm2\") pod \"b453de3f-2c9c-4705-ab30-b3eae093a0b0\" (UID: \"b453de3f-2c9c-4705-ab30-b3eae093a0b0\") " Nov 02 17:57:13.029791 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.029767 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b453de3f-2c9c-4705-ab30-b3eae093a0b0-plugins-config-volume" pod "b453de3f-2c9c-4705-ab30-b3eae093a0b0" (UID: "b453de3f-2c9c-4705-ab30-b3eae093a0b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:13.030141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.030123 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b453de3f-2c9c-4705-ab30-b3eae093a0b0-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:13.032152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.032128 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b453de3f-2c9c-4705-ab30-b3eae093a0b0-kube-api-access-fbhm2" pod "b453de3f-2c9c-4705-ab30-b3eae093a0b0" (UID: "b453de3f-2c9c-4705-ab30-b3eae093a0b0"). InnerVolumeSpecName "kube-api-access-fbhm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:13.147798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.143728 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2469cdd9-c68f-477b-9cb8-85aadba1bacd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2469cdd9-c68f-477b-9cb8-85aadba1bacd-plugins-config-volume\") pod \"2469cdd9-c68f-477b-9cb8-85aadba1bacd\" (UID: \"2469cdd9-c68f-477b-9cb8-85aadba1bacd\") " Nov 02 17:57:13.148206 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.148172 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2469cdd9-c68f-477b-9cb8-85aadba1bacd-kube-api-access-9rccv\" (UniqueName: \"kubernetes.io/projected/2469cdd9-c68f-477b-9cb8-85aadba1bacd-kube-api-access-9rccv\") pod \"2469cdd9-c68f-477b-9cb8-85aadba1bacd\" (UID: \"2469cdd9-c68f-477b-9cb8-85aadba1bacd\") " Nov 02 17:57:13.149179 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.149154 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbhm2\" (UniqueName: \"kubernetes.io/projected/b453de3f-2c9c-4705-ab30-b3eae093a0b0-kube-api-access-fbhm2\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:13.159710 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.158521 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2469cdd9-c68f-477b-9cb8-85aadba1bacd-plugins-config-volume" pod "2469cdd9-c68f-477b-9cb8-85aadba1bacd" (UID: "2469cdd9-c68f-477b-9cb8-85aadba1bacd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:13.348124 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.279614 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:13.348124 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.281201 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2469cdd9-c68f-477b-9cb8-85aadba1bacd-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:13.427143 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.422838 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:13.437500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.427462 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2469cdd9-c68f-477b-9cb8-85aadba1bacd-kube-api-access-9rccv" pod "2469cdd9-c68f-477b-9cb8-85aadba1bacd" (UID: "2469cdd9-c68f-477b-9cb8-85aadba1bacd"). InnerVolumeSpecName "kube-api-access-9rccv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:13.480596 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.480556 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn"] Nov 02 17:57:13.490314 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.490281 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f37d16-678b85bc7c-7fhhn"] Nov 02 17:57:13.528879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.528846 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rccv\" (UniqueName: \"kubernetes.io/projected/2469cdd9-c68f-477b-9cb8-85aadba1bacd-kube-api-access-9rccv\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:13.530218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.530193 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp"] Nov 02 17:57:13.536640 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.536611 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ea3867-g6llp"] Nov 02 17:57:13.579184 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.579150 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:13.716028 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:13.715890 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:14.006377 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.006295 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:14.027884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.027859 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52" Nov 02 17:57:14.055158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.055112 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52"] Nov 02 17:57:14.060490 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.060464 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f37d16-84dcbc8d67-tqb52"] Nov 02 17:57:14.482906 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.482874 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:14.728176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.727430 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2469cdd9-c68f-477b-9cb8-85aadba1bacd" path="/var/lib/kubelet/pods/2469cdd9-c68f-477b-9cb8-85aadba1bacd/volumes" Nov 02 17:57:14.731030 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.728452 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="991b4d35-a453-4122-b3d4-ab356e3304f2" path="/var/lib/kubelet/pods/991b4d35-a453-4122-b3d4-ab356e3304f2/volumes" Nov 02 17:57:14.731030 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.729037 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b453de3f-2c9c-4705-ab30-b3eae093a0b0" path="/var/lib/kubelet/pods/b453de3f-2c9c-4705-ab30-b3eae093a0b0/volumes" Nov 02 17:57:14.777020 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.776994 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:57:14.820978 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.820940 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-plugins-config-volume\") pod \"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a\" (UID: \"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a\") " Nov 02 17:57:14.821188 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.821011 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-kube-api-access-fvrfd\" (UniqueName: \"kubernetes.io/projected/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-kube-api-access-fvrfd\") pod \"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a\" (UID: \"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a\") " Nov 02 17:57:14.821499 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.821463 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-plugins-config-volume" pod "1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a" (UID: "1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:14.821669 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.821648 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:14.823308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.823281 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-kube-api-access-fvrfd" pod "1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a" (UID: "1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a"). InnerVolumeSpecName "kube-api-access-fvrfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:14.922763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:14.922731 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvrfd\" (UniqueName: \"kubernetes.io/projected/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a-kube-api-access-fvrfd\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:15.038513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:15.038419 4310 generic.go:356] "Generic (PLEG): container finished" podID="1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a" containerID="badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f" exitCode=0 Nov 02 17:57:15.038513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:15.038461 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" Nov 02 17:57:15.038738 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:15.038509 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" event={"ID":"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a","Type":"ContainerDied","Data":"badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f"} Nov 02 17:57:15.038738 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:15.038560 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4" event={"ID":"1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a","Type":"ContainerDied","Data":"1decc322f9c7681a1372fd5edbf341d58bf8647cea436afc89ebb34913fd86f5"} Nov 02 17:57:15.038738 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:15.038581 4310 scope.go:122] "RemoveContainer" containerID="badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f" Nov 02 17:57:15.052893 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:15.052868 4310 scope.go:122] "RemoveContainer" containerID="badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f" Nov 02 17:57:15.053281 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:57:15.053260 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f\": container with ID starting with badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f not found: ID does not exist" containerID="badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f" Nov 02 17:57:15.053348 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:15.053289 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f"} err="failed to get container status \"badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f\": rpc error: code = NotFound desc = could not find container \"badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f\": container with ID starting with badae534cc3470bb5d0d027f4eb8f0b18547f39366ca156128b80b5e3fb4937f not found: ID does not exist" Nov 02 17:57:15.068893 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:15.068860 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4"] Nov 02 17:57:15.074362 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:15.074331 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ea3867-59b6fc75bc-7nls4"] Nov 02 17:57:15.209033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:15.208945 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:15.979469 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:15.979436 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:16.675154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:16.675125 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a" path="/var/lib/kubelet/pods/1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a/volumes" Nov 02 17:57:16.683219 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:16.683176 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:17.596772 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:17.596741 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:17.707100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:17.706493 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:17.901326 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:17.901070 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:18.077631 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:18.077563 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:18.102052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:18.102006 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp"] Nov 02 17:57:18.102369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:18.102288 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" containerID="cri-o://907cfa650008c6c144073860e8ff900e24cdb03401a372fb84b6d84d7d92b5fa" gracePeriod=30 Nov 02 17:57:18.110558 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:18.110488 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf"] Nov 02 17:57:18.120353 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:18.117742 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9"] Nov 02 17:57:18.120353 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:18.117953 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" containerID="cri-o://195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72" gracePeriod=30 Nov 02 17:57:18.123855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:18.123823 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq"] Nov 02 17:57:18.124091 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:18.124033 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" containerID="cri-o://7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2" gracePeriod=30 Nov 02 17:57:18.141532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:18.141496 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" containerID="cri-o://d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0" gracePeriod=30 Nov 02 17:57:18.738270 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:18.738240 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:19.578329 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:19.578293 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:19.724195 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:19.723661 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:20.009289 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:20.009253 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:20.111555 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:20.111522 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:20.281141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:20.281040 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:20.825038 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:20.824353 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:21.739680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:21.739652 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:21.918450 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:21.918412 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:22.035145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:22.035048 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:22.372364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:22.372283 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:22.479665 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:22.479624 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:22.772445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:22.772414 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:22.959168 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:22.959133 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:23.231284 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.231242 4310 generic.go:356] "Generic (PLEG): container finished" podID="90556d84-e481-4860-ace0-12f84117cc76" containerID="907cfa650008c6c144073860e8ff900e24cdb03401a372fb84b6d84d7d92b5fa" exitCode=0 Nov 02 17:57:23.231436 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.231297 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" event={"ID":"90556d84-e481-4860-ace0-12f84117cc76","Type":"ContainerDied","Data":"907cfa650008c6c144073860e8ff900e24cdb03401a372fb84b6d84d7d92b5fa"} Nov 02 17:57:23.414845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.414789 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:23.574139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.549938 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:23.574139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.556573 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:57:23.574139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.565187 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:57:23.620691 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.620657 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:57:23.641062 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.641034 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:57:23.746812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.746062 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.753661 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-socket\") pod \"4140a511-0bf5-413c-93d5-f35e2a070523\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.753722 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-credential-socket\") pod \"e62cfdc9-79e3-4568-a547-6caddc29126c\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.753756 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-podinfo\") pod \"a972210a-5a66-4012-9cb4-20120b1b3c21\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.753794 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-kube-api-access-x8zgb\" (UniqueName: \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-kube-api-access-x8zgb\") pod \"e62cfdc9-79e3-4568-a547-6caddc29126c\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.753861 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-istio-token\" (UniqueName: \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-istio-token\") pod \"4140a511-0bf5-413c-93d5-f35e2a070523\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.753901 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4140a511-0bf5-413c-93d5-f35e2a070523-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4140a511-0bf5-413c-93d5-f35e2a070523-istio-podinfo\") pod \"4140a511-0bf5-413c-93d5-f35e2a070523\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.753947 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4140a511-0bf5-413c-93d5-f35e2a070523-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4140a511-0bf5-413c-93d5-f35e2a070523-istiod-ca-cert\") pod \"4140a511-0bf5-413c-93d5-f35e2a070523\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.753994 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-envoy\") pod \"a972210a-5a66-4012-9cb4-20120b1b3c21\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.754032 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-certs\") pod \"a972210a-5a66-4012-9cb4-20120b1b3c21\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.754071 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-data\") pod \"90556d84-e481-4860-ace0-12f84117cc76\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.754117 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-data\") pod \"a972210a-5a66-4012-9cb4-20120b1b3c21\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.754163 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-credential-socket\") pod \"90556d84-e481-4860-ace0-12f84117cc76\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.754201 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-podinfo\") pod \"e62cfdc9-79e3-4568-a547-6caddc29126c\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.754264 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-token\" (UniqueName: \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-token\") pod \"a972210a-5a66-4012-9cb4-20120b1b3c21\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.754327 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-socket\") pod \"a972210a-5a66-4012-9cb4-20120b1b3c21\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.754361 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-envoy\") pod \"e62cfdc9-79e3-4568-a547-6caddc29126c\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757169 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a972210a-5a66-4012-9cb4-20120b1b3c21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a972210a-5a66-4012-9cb4-20120b1b3c21-istiod-ca-cert\") pod \"a972210a-5a66-4012-9cb4-20120b1b3c21\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757231 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-socket\") pod \"e62cfdc9-79e3-4568-a547-6caddc29126c\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757268 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-certs\") pod \"e62cfdc9-79e3-4568-a547-6caddc29126c\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757333 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-certs\") pod \"90556d84-e481-4860-ace0-12f84117cc76\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757371 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-data\") pod \"e62cfdc9-79e3-4568-a547-6caddc29126c\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757437 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-credential-socket\") pod \"a972210a-5a66-4012-9cb4-20120b1b3c21\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757517 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-token\" (UniqueName: \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-token\") pod \"e62cfdc9-79e3-4568-a547-6caddc29126c\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757605 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-data\") pod \"4140a511-0bf5-413c-93d5-f35e2a070523\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757681 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62cfdc9-79e3-4568-a547-6caddc29126c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62cfdc9-79e3-4568-a547-6caddc29126c-istiod-ca-cert\") pod \"e62cfdc9-79e3-4568-a547-6caddc29126c\" (UID: \"e62cfdc9-79e3-4568-a547-6caddc29126c\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757726 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-istio-token\" (UniqueName: \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-istio-token\") pod \"90556d84-e481-4860-ace0-12f84117cc76\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757771 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-certs\") pod \"4140a511-0bf5-413c-93d5-f35e2a070523\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757831 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-kube-api-access-qg8k5\" (UniqueName: \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-kube-api-access-qg8k5\") pod \"4140a511-0bf5-413c-93d5-f35e2a070523\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757866 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-credential-socket\") pod \"4140a511-0bf5-413c-93d5-f35e2a070523\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757920 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90556d84-e481-4860-ace0-12f84117cc76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90556d84-e481-4860-ace0-12f84117cc76-istiod-ca-cert\") pod \"90556d84-e481-4860-ace0-12f84117cc76\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757968 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90556d84-e481-4860-ace0-12f84117cc76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90556d84-e481-4860-ace0-12f84117cc76-istio-podinfo\") pod \"90556d84-e481-4860-ace0-12f84117cc76\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.757996 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-envoy\") pod \"4140a511-0bf5-413c-93d5-f35e2a070523\" (UID: \"4140a511-0bf5-413c-93d5-f35e2a070523\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.758041 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-socket\") pod \"90556d84-e481-4860-ace0-12f84117cc76\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.758130 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-kube-api-access-hz4vf\" (UniqueName: \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-kube-api-access-hz4vf\") pod \"90556d84-e481-4860-ace0-12f84117cc76\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.758176 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-envoy\") pod \"90556d84-e481-4860-ace0-12f84117cc76\" (UID: \"90556d84-e481-4860-ace0-12f84117cc76\") " Nov 02 17:57:23.760114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.758228 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-kube-api-access-74rpb\" (UniqueName: \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-kube-api-access-74rpb\") pod \"a972210a-5a66-4012-9cb4-20120b1b3c21\" (UID: \"a972210a-5a66-4012-9cb4-20120b1b3c21\") " Nov 02 17:57:23.762209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.761523 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-certs" pod "4140a511-0bf5-413c-93d5-f35e2a070523" (UID: "4140a511-0bf5-413c-93d5-f35e2a070523"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.762209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.761747 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-certs" pod "e62cfdc9-79e3-4568-a547-6caddc29126c" (UID: "e62cfdc9-79e3-4568-a547-6caddc29126c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.762209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.761974 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-certs" pod "90556d84-e481-4860-ace0-12f84117cc76" (UID: "90556d84-e481-4860-ace0-12f84117cc76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.762209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.762061 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-credential-socket" pod "4140a511-0bf5-413c-93d5-f35e2a070523" (UID: "4140a511-0bf5-413c-93d5-f35e2a070523"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.762209 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.762195 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-data" pod "e62cfdc9-79e3-4568-a547-6caddc29126c" (UID: "e62cfdc9-79e3-4568-a547-6caddc29126c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.762457 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.762417 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-credential-socket" pod "a972210a-5a66-4012-9cb4-20120b1b3c21" (UID: "a972210a-5a66-4012-9cb4-20120b1b3c21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.766512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.763091 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-certs" pod "a972210a-5a66-4012-9cb4-20120b1b3c21" (UID: "a972210a-5a66-4012-9cb4-20120b1b3c21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.766512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.766458 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx"] Nov 02 17:57:23.768315 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.768281 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-istio-token" pod "4140a511-0bf5-413c-93d5-f35e2a070523" (UID: "4140a511-0bf5-413c-93d5-f35e2a070523"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.770565 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.770532 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-socket" pod "e62cfdc9-79e3-4568-a547-6caddc29126c" (UID: "e62cfdc9-79e3-4568-a547-6caddc29126c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.771302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.771099 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-podinfo" pod "e62cfdc9-79e3-4568-a547-6caddc29126c" (UID: "e62cfdc9-79e3-4568-a547-6caddc29126c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:23.771625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.771446 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-data" pod "90556d84-e481-4860-ace0-12f84117cc76" (UID: "90556d84-e481-4860-ace0-12f84117cc76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.771953 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.771765 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-data" pod "a972210a-5a66-4012-9cb4-20120b1b3c21" (UID: "a972210a-5a66-4012-9cb4-20120b1b3c21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.772141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.772091 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-credential-socket" pod "90556d84-e481-4860-ace0-12f84117cc76" (UID: "90556d84-e481-4860-ace0-12f84117cc76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.772560 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.772538 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4140a511-0bf5-413c-93d5-f35e2a070523-istio-podinfo" pod "4140a511-0bf5-413c-93d5-f35e2a070523" (UID: "4140a511-0bf5-413c-93d5-f35e2a070523"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:23.773601 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.773309 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4140a511-0bf5-413c-93d5-f35e2a070523-istiod-ca-cert" pod "4140a511-0bf5-413c-93d5-f35e2a070523" (UID: "4140a511-0bf5-413c-93d5-f35e2a070523"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:23.775413 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.775376 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-socket" pod "90556d84-e481-4860-ace0-12f84117cc76" (UID: "90556d84-e481-4860-ace0-12f84117cc76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.775672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.775645 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-socket" pod "4140a511-0bf5-413c-93d5-f35e2a070523" (UID: "4140a511-0bf5-413c-93d5-f35e2a070523"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.775863 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.775841 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-credential-socket" pod "e62cfdc9-79e3-4568-a547-6caddc29126c" (UID: "e62cfdc9-79e3-4568-a547-6caddc29126c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.776476 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.776445 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-data" pod "4140a511-0bf5-413c-93d5-f35e2a070523" (UID: "4140a511-0bf5-413c-93d5-f35e2a070523"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.777617 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.776695 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-envoy" pod "a972210a-5a66-4012-9cb4-20120b1b3c21" (UID: "a972210a-5a66-4012-9cb4-20120b1b3c21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.777617 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.776734 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62cfdc9-79e3-4568-a547-6caddc29126c-istiod-ca-cert" pod "e62cfdc9-79e3-4568-a547-6caddc29126c" (UID: "e62cfdc9-79e3-4568-a547-6caddc29126c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:23.778130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.777924 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-kube-api-access-hz4vf" pod "90556d84-e481-4860-ace0-12f84117cc76" (UID: "90556d84-e481-4860-ace0-12f84117cc76"). InnerVolumeSpecName "kube-api-access-hz4vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.779246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.778297 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" containerID="cri-o://3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5" gracePeriod=30 Nov 02 17:57:23.779246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.778388 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90556d84-e481-4860-ace0-12f84117cc76-istiod-ca-cert" pod "90556d84-e481-4860-ace0-12f84117cc76" (UID: "90556d84-e481-4860-ace0-12f84117cc76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:23.780849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.780680 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-socket" pod "a972210a-5a66-4012-9cb4-20120b1b3c21" (UID: "a972210a-5a66-4012-9cb4-20120b1b3c21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.783018 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.782976 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng"] Nov 02 17:57:23.783831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.783517 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" containerID="cri-o://14715dacc085e0427bb077b4ae06842eaefb155c908617b2ad4650f36506c2df" gracePeriod=30 Nov 02 17:57:23.784662 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.784465 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a972210a-5a66-4012-9cb4-20120b1b3c21-istiod-ca-cert" pod "a972210a-5a66-4012-9cb4-20120b1b3c21" (UID: "a972210a-5a66-4012-9cb4-20120b1b3c21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:23.789273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.789237 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-istio-token" pod "90556d84-e481-4860-ace0-12f84117cc76" (UID: "90556d84-e481-4860-ace0-12f84117cc76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.789371 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.789351 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-podinfo" pod "a972210a-5a66-4012-9cb4-20120b1b3c21" (UID: "a972210a-5a66-4012-9cb4-20120b1b3c21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:23.789487 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.789465 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-kube-api-access-qg8k5" pod "4140a511-0bf5-413c-93d5-f35e2a070523" (UID: "4140a511-0bf5-413c-93d5-f35e2a070523"). InnerVolumeSpecName "kube-api-access-qg8k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.789554 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.789531 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-envoy" pod "4140a511-0bf5-413c-93d5-f35e2a070523" (UID: "4140a511-0bf5-413c-93d5-f35e2a070523"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.789701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.789611 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-token" pod "e62cfdc9-79e3-4568-a547-6caddc29126c" (UID: "e62cfdc9-79e3-4568-a547-6caddc29126c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.789701 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.789679 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-kube-api-access-x8zgb" pod "e62cfdc9-79e3-4568-a547-6caddc29126c" (UID: "e62cfdc9-79e3-4568-a547-6caddc29126c"). InnerVolumeSpecName "kube-api-access-x8zgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.790302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.789756 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-envoy" pod "e62cfdc9-79e3-4568-a547-6caddc29126c" (UID: "e62cfdc9-79e3-4568-a547-6caddc29126c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.790302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.789889 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-envoy" pod "90556d84-e481-4860-ace0-12f84117cc76" (UID: "90556d84-e481-4860-ace0-12f84117cc76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:23.798681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.798132 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-token" pod "a972210a-5a66-4012-9cb4-20120b1b3c21" (UID: "a972210a-5a66-4012-9cb4-20120b1b3c21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.798681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.798166 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90556d84-e481-4860-ace0-12f84117cc76-istio-podinfo" pod "90556d84-e481-4860-ace0-12f84117cc76" (UID: "90556d84-e481-4860-ace0-12f84117cc76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:23.799735 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.799483 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-kube-api-access-74rpb" pod "a972210a-5a66-4012-9cb4-20120b1b3c21" (UID: "a972210a-5a66-4012-9cb4-20120b1b3c21"). InnerVolumeSpecName "kube-api-access-74rpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:23.810149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.810124 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz"] Nov 02 17:57:23.810370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.810349 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" containerID="cri-o://dd4e4a1680024ff19d22cedf70a187a33d07dced10cd875cb494079d2d7a98c2" gracePeriod=30 Nov 02 17:57:23.816157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.816133 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd"] Nov 02 17:57:23.816368 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.816346 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" containerID="cri-o://740b0c0155ff13bdb5d7b597ca33fdb3baaa1933df177fdda2a156fcfc7e30dd" gracePeriod=30 Nov 02 17:57:23.857129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.857098 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.860860 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.860892 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.860908 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a972210a-5a66-4012-9cb4-20120b1b3c21-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.860922 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.860935 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.860950 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.860964 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.860979 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.860993 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861006 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861021 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62cfdc9-79e3-4568-a547-6caddc29126c-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861033 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861047 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861060 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg8k5\" (UniqueName: \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-kube-api-access-qg8k5\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861073 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861105 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90556d84-e481-4860-ace0-12f84117cc76-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861120 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90556d84-e481-4860-ace0-12f84117cc76-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861134 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861148 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861162 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz4vf\" (UniqueName: \"kubernetes.io/projected/90556d84-e481-4860-ace0-12f84117cc76-kube-api-access-hz4vf\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861177 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861190 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74rpb\" (UniqueName: \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-kube-api-access-74rpb\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861205 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4140a511-0bf5-413c-93d5-f35e2a070523-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861218 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62cfdc9-79e3-4568-a547-6caddc29126c-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861234 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861246 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8zgb\" (UniqueName: \"kubernetes.io/projected/e62cfdc9-79e3-4568-a547-6caddc29126c-kube-api-access-x8zgb\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861259 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4140a511-0bf5-413c-93d5-f35e2a070523-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861272 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4140a511-0bf5-413c-93d5-f35e2a070523-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861285 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4140a511-0bf5-413c-93d5-f35e2a070523-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861297 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861309 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861323 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861335 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861348 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90556d84-e481-4860-ace0-12f84117cc76-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861361 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62cfdc9-79e3-4568-a547-6caddc29126c-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:23.862855 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:23.861375 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a972210a-5a66-4012-9cb4-20120b1b3c21-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:24.241584 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.241547 4310 generic.go:356] "Generic (PLEG): container finished" podID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerID="7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2" exitCode=0 Nov 02 17:57:24.241731 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.241623 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" Nov 02 17:57:24.241731 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.241633 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" event={"ID":"e62cfdc9-79e3-4568-a547-6caddc29126c","Type":"ContainerDied","Data":"7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2"} Nov 02 17:57:24.241731 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.241690 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq" event={"ID":"e62cfdc9-79e3-4568-a547-6caddc29126c","Type":"ContainerDied","Data":"ce5a4b6851baa6b0a75e28d4da42bd4916029844e0ad50cd2d49c2097ba90c7f"} Nov 02 17:57:24.241731 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.241720 4310 scope.go:122] "RemoveContainer" containerID="7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2" Nov 02 17:57:24.243505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.243483 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" Nov 02 17:57:24.243505 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.243495 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp" event={"ID":"90556d84-e481-4860-ace0-12f84117cc76","Type":"ContainerDied","Data":"0f2fa2cf2f808835a9c1248042b7f85bda75b9b4974005d5995c522ff283aa71"} Nov 02 17:57:24.245310 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.245279 4310 generic.go:356] "Generic (PLEG): container finished" podID="4140a511-0bf5-413c-93d5-f35e2a070523" containerID="195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72" exitCode=0 Nov 02 17:57:24.245445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.245353 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" Nov 02 17:57:24.245445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.245374 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" event={"ID":"4140a511-0bf5-413c-93d5-f35e2a070523","Type":"ContainerDied","Data":"195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72"} Nov 02 17:57:24.245445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.245427 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9" event={"ID":"4140a511-0bf5-413c-93d5-f35e2a070523","Type":"ContainerDied","Data":"2f1ab95d50c7132fbb26fd97e83bf4951a7de130ee86ca0b0693e771c61baf9a"} Nov 02 17:57:24.247327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.247303 4310 generic.go:356] "Generic (PLEG): container finished" podID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerID="d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0" exitCode=0 Nov 02 17:57:24.247452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.247352 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" Nov 02 17:57:24.247452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.247383 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" event={"ID":"a972210a-5a66-4012-9cb4-20120b1b3c21","Type":"ContainerDied","Data":"d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0"} Nov 02 17:57:24.247452 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.247433 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf" event={"ID":"a972210a-5a66-4012-9cb4-20120b1b3c21","Type":"ContainerDied","Data":"36fd146327be73a2a2768a7a0e009b40cdc4bc44567f38663a73e41b7d341c75"} Nov 02 17:57:24.257022 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.256831 4310 scope.go:122] "RemoveContainer" containerID="7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2" Nov 02 17:57:24.257206 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:57:24.257182 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2\": container with ID starting with 7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2 not found: ID does not exist" containerID="7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2" Nov 02 17:57:24.257273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.257221 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2"} err="failed to get container status \"7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2\": rpc error: code = NotFound desc = could not find container \"7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2\": container with ID starting with 7137bd716c4f113aec6d0959cd7791bf6c02ee3416966690ce04880ccf61acc2 not found: ID does not exist" Nov 02 17:57:24.257273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.257246 4310 scope.go:122] "RemoveContainer" containerID="907cfa650008c6c144073860e8ff900e24cdb03401a372fb84b6d84d7d92b5fa" Nov 02 17:57:24.269231 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.269212 4310 scope.go:122] "RemoveContainer" containerID="195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72" Nov 02 17:57:24.278059 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.278033 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq"] Nov 02 17:57:24.280296 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.280279 4310 scope.go:122] "RemoveContainer" containerID="195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72" Nov 02 17:57:24.280611 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:57:24.280592 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72\": container with ID starting with 195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72 not found: ID does not exist" containerID="195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72" Nov 02 17:57:24.280676 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.280621 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72"} err="failed to get container status \"195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72\": rpc error: code = NotFound desc = could not find container \"195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72\": container with ID starting with 195e604e176287f19944bfa89f818890ba8af9c576ca29d322851dd3966c9b72 not found: ID does not exist" Nov 02 17:57:24.280676 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.280642 4310 scope.go:122] "RemoveContainer" containerID="d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0" Nov 02 17:57:24.283920 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.283861 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-pgstq"] Nov 02 17:57:24.289795 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.289771 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9"] Nov 02 17:57:24.292879 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.292865 4310 scope.go:122] "RemoveContainer" containerID="d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0" Nov 02 17:57:24.293168 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:57:24.293148 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0\": container with ID starting with d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0 not found: ID does not exist" containerID="d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0" Nov 02 17:57:24.293242 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.293173 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0"} err="failed to get container status \"d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0\": rpc error: code = NotFound desc = could not find container \"d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0\": container with ID starting with d2bdb5222e6b20626cc7ff237eeddd86fa81fafdd3e3b8eac7f949440d0732b0 not found: ID does not exist" Nov 02 17:57:24.294786 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.294768 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-qjvl9"] Nov 02 17:57:24.306033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.306013 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf"] Nov 02 17:57:24.310367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.310349 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-wr6xf"] Nov 02 17:57:24.320860 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.320834 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp"] Nov 02 17:57:24.325425 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.325404 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f37d16-openshift-default-75474b5775-p5bzp"] Nov 02 17:57:24.586899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.586861 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:24.708108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.703151 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" path="/var/lib/kubelet/pods/4140a511-0bf5-413c-93d5-f35e2a070523/volumes" Nov 02 17:57:24.708108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.703840 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90556d84-e481-4860-ace0-12f84117cc76" path="/var/lib/kubelet/pods/90556d84-e481-4860-ace0-12f84117cc76/volumes" Nov 02 17:57:24.708108 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.707635 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" path="/var/lib/kubelet/pods/a972210a-5a66-4012-9cb4-20120b1b3c21/volumes" Nov 02 17:57:24.710295 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:24.708575 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" path="/var/lib/kubelet/pods/e62cfdc9-79e3-4568-a547-6caddc29126c/volumes" Nov 02 17:57:25.176337 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.176288 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:25.319137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319104 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz"] Nov 02 17:57:25.319812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319789 4310 cpu_manager.go:414] "containerMap: removing container" podUID="b453de3f-2c9c-4705-ab30-b3eae093a0b0" containerName="payload-processing" Nov 02 17:57:25.319812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319810 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b453de3f-2c9c-4705-ab30-b3eae093a0b0" containerName="payload-processing" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319819 4310 cpu_manager.go:414] "containerMap: removing container" podUID="1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a" containerName="payload-pre-processing" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319824 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a" containerName="payload-pre-processing" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319838 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319843 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319850 4310 cpu_manager.go:414] "containerMap: removing container" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319855 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319866 4310 cpu_manager.go:414] "containerMap: removing container" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319870 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319879 4310 cpu_manager.go:414] "containerMap: removing container" podUID="6e328e69-eeb5-4258-81c3-7a56397c0563" containerName="revoke-keys" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319884 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e328e69-eeb5-4258-81c3-7a56397c0563" containerName="revoke-keys" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319893 4310 cpu_manager.go:414] "containerMap: removing container" podUID="2469cdd9-c68f-477b-9cb8-85aadba1bacd" containerName="payload-pre-processing" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319898 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2469cdd9-c68f-477b-9cb8-85aadba1bacd" containerName="payload-pre-processing" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319913 4310 cpu_manager.go:414] "containerMap: removing container" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" Nov 02 17:57:25.319966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319917 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" Nov 02 17:57:25.320751 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.319996 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="90556d84-e481-4860-ace0-12f84117cc76" containerName="istio-proxy" Nov 02 17:57:25.320751 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.320007 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62cfdc9-79e3-4568-a547-6caddc29126c" containerName="istio-proxy" Nov 02 17:57:25.320751 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.320014 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e328e69-eeb5-4258-81c3-7a56397c0563" containerName="revoke-keys" Nov 02 17:57:25.320751 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.320020 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="2469cdd9-c68f-477b-9cb8-85aadba1bacd" containerName="payload-pre-processing" Nov 02 17:57:25.320751 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.320028 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="4140a511-0bf5-413c-93d5-f35e2a070523" containerName="istio-proxy" Nov 02 17:57:25.320751 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.320037 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="b453de3f-2c9c-4705-ab30-b3eae093a0b0" containerName="payload-processing" Nov 02 17:57:25.320751 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.320042 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3ad3de-693e-4371-8e24-cc5d3d3e7f6a" containerName="payload-pre-processing" Nov 02 17:57:25.320751 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.320049 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a972210a-5a66-4012-9cb4-20120b1b3c21" containerName="istio-proxy" Nov 02 17:57:25.325675 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.325655 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:25.330388 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.330368 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:57:25.346291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.346265 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz"] Nov 02 17:57:25.379450 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.379387 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:25.428562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.428456 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:25.428743 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.428669 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdrd6\" (UniqueName: \"kubernetes.io/projected/a2d547b3-05f3-4108-a9df-272a923909f9-kube-api-access-gdrd6\") pod \"maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:25.428743 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.428706 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:25.431494 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.431322 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:57:25.530297 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.530264 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:25.530445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.530365 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdrd6\" (UniqueName: \"kubernetes.io/projected/a2d547b3-05f3-4108-a9df-272a923909f9-kube-api-access-gdrd6\") pod \"maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:25.530445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.530388 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:25.531073 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.531045 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:25.532742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.532717 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:25.543554 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.543531 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdrd6\" (UniqueName: \"kubernetes.io/projected/a2d547b3-05f3-4108-a9df-272a923909f9-kube-api-access-gdrd6\") pod \"maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:25.641124 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.641053 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:25.945135 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:57:25.945056 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d547b3_05f3_4108_a9df_272a923909f9.slice/crio-717279fbf6d8582aaa5510326f2de3e1d8a76c4792de2a76b41ffbe326b7b022 WatchSource:0}: Error finding container 717279fbf6d8582aaa5510326f2de3e1d8a76c4792de2a76b41ffbe326b7b022: Status 404 returned error can't find the container with id 717279fbf6d8582aaa5510326f2de3e1d8a76c4792de2a76b41ffbe326b7b022 Nov 02 17:57:25.946332 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:25.946308 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz"] Nov 02 17:57:26.273162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:26.273124 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" event={"ID":"a2d547b3-05f3-4108-a9df-272a923909f9","Type":"ContainerStarted","Data":"41b033bf84ef4d24c01ede05e3dfd0162a345df3df613307eca344b44e44a2d8"} Nov 02 17:57:26.273509 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:26.273180 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" event={"ID":"a2d547b3-05f3-4108-a9df-272a923909f9","Type":"ContainerStarted","Data":"717279fbf6d8582aaa5510326f2de3e1d8a76c4792de2a76b41ffbe326b7b022"} Nov 02 17:57:26.298716 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:26.298665 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" podStartSLOduration=1.298651061 podStartE2EDuration="1.298651061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:26.292835503 +0000 UTC m=+1043.735918140" watchObservedRunningTime="2026-11-02 17:57:26.298651061 +0000 UTC m=+1043.741733720" Nov 02 17:57:26.391862 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:26.391826 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:27.311009 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:27.310961 4310 generic.go:356] "Generic (PLEG): container finished" podID="a2d547b3-05f3-4108-a9df-272a923909f9" containerID="41b033bf84ef4d24c01ede05e3dfd0162a345df3df613307eca344b44e44a2d8" exitCode=0 Nov 02 17:57:27.311827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:27.311792 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" event={"ID":"a2d547b3-05f3-4108-a9df-272a923909f9","Type":"ContainerDied","Data":"41b033bf84ef4d24c01ede05e3dfd0162a345df3df613307eca344b44e44a2d8"} Nov 02 17:57:27.572889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:27.572786 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:28.912058 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:28.862588 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:28.969204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:28.969167 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:29.114180 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.111404 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:29.163609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.163575 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:57:29.163776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.163633 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:57:29.209446 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.209052 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:57:29.225715 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.225675 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d547b3-05f3-4108-a9df-272a923909f9-kube-api-access-gdrd6\" (UniqueName: \"kubernetes.io/projected/a2d547b3-05f3-4108-a9df-272a923909f9-kube-api-access-gdrd6\") pod \"a2d547b3-05f3-4108-a9df-272a923909f9\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " Nov 02 17:57:29.226173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.226028 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-xks-ca\") pod \"a2d547b3-05f3-4108-a9df-272a923909f9\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " Nov 02 17:57:29.226173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.226114 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-openshift-ca\") pod \"a2d547b3-05f3-4108-a9df-272a923909f9\" (UID: \"a2d547b3-05f3-4108-a9df-272a923909f9\") " Nov 02 17:57:29.226845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.226653 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-openshift-ca" pod "a2d547b3-05f3-4108-a9df-272a923909f9" (UID: "a2d547b3-05f3-4108-a9df-272a923909f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:29.227042 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.227019 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.233246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.233211 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-xks-ca" pod "a2d547b3-05f3-4108-a9df-272a923909f9" (UID: "a2d547b3-05f3-4108-a9df-272a923909f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:29.252864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.252807 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d547b3-05f3-4108-a9df-272a923909f9-kube-api-access-gdrd6" pod "a2d547b3-05f3-4108-a9df-272a923909f9" (UID: "a2d547b3-05f3-4108-a9df-272a923909f9"). InnerVolumeSpecName "kube-api-access-gdrd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.328180 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-podinfo\") pod \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.328270 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-token\" (UniqueName: \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-token\") pod \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.328349 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-certs\") pod \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.328404 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-credential-socket\") pod \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.328565 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istiod-ca-cert\") pod \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.328657 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-kube-api-access-rbflh\" (UniqueName: \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-kube-api-access-rbflh\") pod \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.328747 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-envoy\") pod \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.328800 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-data\") pod \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.328909 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-socket\") pod \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\" (UID: \"b2997e8d-e663-4e09-b1ea-a2a796d00aaa\") " Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.329777 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d547b3-05f3-4108-a9df-272a923909f9-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.329801 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdrd6\" (UniqueName: \"kubernetes.io/projected/a2d547b3-05f3-4108-a9df-272a923909f9-kube-api-access-gdrd6\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.330113 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-socket" pod "b2997e8d-e663-4e09-b1ea-a2a796d00aaa" (UID: "b2997e8d-e663-4e09-b1ea-a2a796d00aaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.330406 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-certs" pod "b2997e8d-e663-4e09-b1ea-a2a796d00aaa" (UID: "b2997e8d-e663-4e09-b1ea-a2a796d00aaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.330771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.330633 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-credential-socket" pod "b2997e8d-e663-4e09-b1ea-a2a796d00aaa" (UID: "b2997e8d-e663-4e09-b1ea-a2a796d00aaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.331687 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.330955 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istiod-ca-cert" pod "b2997e8d-e663-4e09-b1ea-a2a796d00aaa" (UID: "b2997e8d-e663-4e09-b1ea-a2a796d00aaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:29.332409 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.332305 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-data" pod "b2997e8d-e663-4e09-b1ea-a2a796d00aaa" (UID: "b2997e8d-e663-4e09-b1ea-a2a796d00aaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.333577 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.333540 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-token" pod "b2997e8d-e663-4e09-b1ea-a2a796d00aaa" (UID: "b2997e8d-e663-4e09-b1ea-a2a796d00aaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.335198 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.335154 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-podinfo" pod "b2997e8d-e663-4e09-b1ea-a2a796d00aaa" (UID: "b2997e8d-e663-4e09-b1ea-a2a796d00aaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:29.336769 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.336716 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-envoy" pod "b2997e8d-e663-4e09-b1ea-a2a796d00aaa" (UID: "b2997e8d-e663-4e09-b1ea-a2a796d00aaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.337002 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.336960 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-kube-api-access-rbflh" pod "b2997e8d-e663-4e09-b1ea-a2a796d00aaa" (UID: "b2997e8d-e663-4e09-b1ea-a2a796d00aaa"). InnerVolumeSpecName "kube-api-access-rbflh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.397309 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.396976 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:29.424475 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.424429 4310 generic.go:356] "Generic (PLEG): container finished" podID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerID="14715dacc085e0427bb077b4ae06842eaefb155c908617b2ad4650f36506c2df" exitCode=0 Nov 02 17:57:29.424625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.424554 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" event={"ID":"58e331e1-a05d-491b-a48c-913b3bb8f45e","Type":"ContainerDied","Data":"14715dacc085e0427bb077b4ae06842eaefb155c908617b2ad4650f36506c2df"} Nov 02 17:57:29.426572 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.426549 4310 generic.go:356] "Generic (PLEG): container finished" podID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerID="3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5" exitCode=0 Nov 02 17:57:29.426719 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.426704 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" event={"ID":"b2997e8d-e663-4e09-b1ea-a2a796d00aaa","Type":"ContainerDied","Data":"3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5"} Nov 02 17:57:29.426789 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.426737 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" event={"ID":"b2997e8d-e663-4e09-b1ea-a2a796d00aaa","Type":"ContainerDied","Data":"c801ba8ddd676d7987deb631f88ceec35661d8f7bb5a11a033ae70eff8e051af"} Nov 02 17:57:29.426789 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.426759 4310 scope.go:122] "RemoveContainer" containerID="3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5" Nov 02 17:57:29.427195 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.427179 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.429962 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" event={"ID":"a2d547b3-05f3-4108-a9df-272a923909f9","Type":"ContainerDied","Data":"717279fbf6d8582aaa5510326f2de3e1d8a76c4792de2a76b41ffbe326b7b022"} Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.430009 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="717279fbf6d8582aaa5510326f2de3e1d8a76c4792de2a76b41ffbe326b7b022" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.430064 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.430763 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.431934 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.431956 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.431970 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.431983 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.431997 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.432011 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.432026 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbflh\" (UniqueName: \"kubernetes.io/projected/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-kube-api-access-rbflh\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.432041 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.433057 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.432055 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2997e8d-e663-4e09-b1ea-a2a796d00aaa-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.437208 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.435715 4310 generic.go:356] "Generic (PLEG): container finished" podID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerID="dd4e4a1680024ff19d22cedf70a187a33d07dced10cd875cb494079d2d7a98c2" exitCode=0 Nov 02 17:57:29.437208 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.435882 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" event={"ID":"d23349c1-0d41-41eb-a39a-5e7c1f034b5a","Type":"ContainerDied","Data":"dd4e4a1680024ff19d22cedf70a187a33d07dced10cd875cb494079d2d7a98c2"} Nov 02 17:57:29.445550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.439714 4310 generic.go:356] "Generic (PLEG): container finished" podID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerID="740b0c0155ff13bdb5d7b597ca33fdb3baaa1933df177fdda2a156fcfc7e30dd" exitCode=0 Nov 02 17:57:29.445550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.439745 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" event={"ID":"7dcdd0a3-0820-4007-977e-c0595b979cb7","Type":"ContainerDied","Data":"740b0c0155ff13bdb5d7b597ca33fdb3baaa1933df177fdda2a156fcfc7e30dd"} Nov 02 17:57:29.445550 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.445360 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:57:29.446346 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.446070 4310 scope.go:122] "RemoveContainer" containerID="3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5" Nov 02 17:57:29.446898 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:57:29.446748 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5\": container with ID starting with 3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5 not found: ID does not exist" containerID="3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5" Nov 02 17:57:29.446898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.446786 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5"} err="failed to get container status \"3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5\": rpc error: code = NotFound desc = could not find container \"3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5\": container with ID starting with 3124cf93160b40814bf2dac0080777ac46465a7e6ebabb1d84c1b5613fd3e6f5 not found: ID does not exist" Nov 02 17:57:29.446898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.446810 4310 scope.go:122] "RemoveContainer" containerID="dd4e4a1680024ff19d22cedf70a187a33d07dced10cd875cb494079d2d7a98c2" Nov 02 17:57:29.459445 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.459365 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:57:29.494116 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.493747 4310 scope.go:122] "RemoveContainer" containerID="18a529e39ff27b70b8e6c851b9371f6e245415d73e6fe9d0f7ff3b379fcf9df9" Nov 02 17:57:29.532573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.532538 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-token\" (UniqueName: \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-token\") pod \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " Nov 02 17:57:29.532812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.532790 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-certs\") pod \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " Nov 02 17:57:29.532981 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.532965 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-token\" (UniqueName: \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-token\") pod \"7dcdd0a3-0820-4007-977e-c0595b979cb7\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " Nov 02 17:57:29.533137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.533104 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-kube-api-access-hlddz\" (UniqueName: \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-kube-api-access-hlddz\") pod \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " Nov 02 17:57:29.533348 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.533330 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dcdd0a3-0820-4007-977e-c0595b979cb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcdd0a3-0820-4007-977e-c0595b979cb7-istiod-ca-cert\") pod \"7dcdd0a3-0820-4007-977e-c0595b979cb7\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " Nov 02 17:57:29.533474 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.533458 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e331e1-a05d-491b-a48c-913b3bb8f45e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e331e1-a05d-491b-a48c-913b3bb8f45e-istiod-ca-cert\") pod \"58e331e1-a05d-491b-a48c-913b3bb8f45e\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " Nov 02 17:57:29.533596 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.533582 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-envoy\") pod \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " Nov 02 17:57:29.533729 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.533714 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-certs\") pod \"58e331e1-a05d-491b-a48c-913b3bb8f45e\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " Nov 02 17:57:29.533831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.533137 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-certs" pod "d23349c1-0d41-41eb-a39a-5e7c1f034b5a" (UID: "d23349c1-0d41-41eb-a39a-5e7c1f034b5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.533965 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.533799 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dcdd0a3-0820-4007-977e-c0595b979cb7-istiod-ca-cert" pod "7dcdd0a3-0820-4007-977e-c0595b979cb7" (UID: "7dcdd0a3-0820-4007-977e-c0595b979cb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:29.534218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.533842 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e331e1-a05d-491b-a48c-913b3bb8f45e-istiod-ca-cert" pod "58e331e1-a05d-491b-a48c-913b3bb8f45e" (UID: "58e331e1-a05d-491b-a48c-913b3bb8f45e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:29.534302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.534054 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-certs" pod "7dcdd0a3-0820-4007-977e-c0595b979cb7" (UID: "7dcdd0a3-0820-4007-977e-c0595b979cb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.534385 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.534297 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-certs" pod "58e331e1-a05d-491b-a48c-913b3bb8f45e" (UID: "58e331e1-a05d-491b-a48c-913b3bb8f45e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.534462 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.533826 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-certs\") pod \"7dcdd0a3-0820-4007-977e-c0595b979cb7\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " Nov 02 17:57:29.534588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.534571 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-kube-api-access-tvcw7\" (UniqueName: \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-kube-api-access-tvcw7\") pod \"7dcdd0a3-0820-4007-977e-c0595b979cb7\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " Nov 02 17:57:29.534709 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.534694 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-envoy\") pod \"7dcdd0a3-0820-4007-977e-c0595b979cb7\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " Nov 02 17:57:29.534838 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.534822 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istiod-ca-cert\") pod \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " Nov 02 17:57:29.535034 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.535017 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-data\") pod \"7dcdd0a3-0820-4007-977e-c0595b979cb7\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " Nov 02 17:57:29.535222 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.535205 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-podinfo\") pod \"58e331e1-a05d-491b-a48c-913b3bb8f45e\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " Nov 02 17:57:29.535355 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.535340 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-kube-api-access-fhp4g\" (UniqueName: \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-kube-api-access-fhp4g\") pod \"58e331e1-a05d-491b-a48c-913b3bb8f45e\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " Nov 02 17:57:29.535459 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.535446 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-data\") pod \"58e331e1-a05d-491b-a48c-913b3bb8f45e\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " Nov 02 17:57:29.535590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.535574 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-token\" (UniqueName: \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-token\") pod \"58e331e1-a05d-491b-a48c-913b3bb8f45e\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " Nov 02 17:57:29.535729 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.535716 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-podinfo\") pod \"7dcdd0a3-0820-4007-977e-c0595b979cb7\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " Nov 02 17:57:29.535843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.535829 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-credential-socket\") pod \"58e331e1-a05d-491b-a48c-913b3bb8f45e\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " Nov 02 17:57:29.535972 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.535958 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-socket\") pod \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " Nov 02 17:57:29.536130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.536113 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-data\") pod \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " Nov 02 17:57:29.536307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.536292 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-envoy\") pod \"58e331e1-a05d-491b-a48c-913b3bb8f45e\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " Nov 02 17:57:29.536430 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.536416 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-socket\") pod \"58e331e1-a05d-491b-a48c-913b3bb8f45e\" (UID: \"58e331e1-a05d-491b-a48c-913b3bb8f45e\") " Nov 02 17:57:29.536564 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.536548 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-credential-socket\") pod \"7dcdd0a3-0820-4007-977e-c0595b979cb7\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " Nov 02 17:57:29.536743 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.536727 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-podinfo\") pod \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " Nov 02 17:57:29.536899 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.536885 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-socket\") pod \"7dcdd0a3-0820-4007-977e-c0595b979cb7\" (UID: \"7dcdd0a3-0820-4007-977e-c0595b979cb7\") " Nov 02 17:57:29.537051 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.537037 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-credential-socket\") pod \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\" (UID: \"d23349c1-0d41-41eb-a39a-5e7c1f034b5a\") " Nov 02 17:57:29.537598 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.537571 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-credential-socket" pod "58e331e1-a05d-491b-a48c-913b3bb8f45e" (UID: "58e331e1-a05d-491b-a48c-913b3bb8f45e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.537901 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.537877 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istiod-ca-cert" pod "d23349c1-0d41-41eb-a39a-5e7c1f034b5a" (UID: "d23349c1-0d41-41eb-a39a-5e7c1f034b5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:29.538151 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.538129 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-data" pod "7dcdd0a3-0820-4007-977e-c0595b979cb7" (UID: "7dcdd0a3-0820-4007-977e-c0595b979cb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.538260 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.538242 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.538369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.538355 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcdd0a3-0820-4007-977e-c0595b979cb7-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.538461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.538449 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e331e1-a05d-491b-a48c-913b3bb8f45e-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.538548 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.538536 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.538633 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.538621 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.538716 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.538704 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.538805 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.538794 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.538878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.538306 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-envoy" pod "7dcdd0a3-0820-4007-977e-c0595b979cb7" (UID: "7dcdd0a3-0820-4007-977e-c0595b979cb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.538951 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.538498 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-data" pod "58e331e1-a05d-491b-a48c-913b3bb8f45e" (UID: "58e331e1-a05d-491b-a48c-913b3bb8f45e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.539039 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.538649 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-kube-api-access-tvcw7" pod "7dcdd0a3-0820-4007-977e-c0595b979cb7" (UID: "7dcdd0a3-0820-4007-977e-c0595b979cb7"). InnerVolumeSpecName "kube-api-access-tvcw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.539342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.539323 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-socket" pod "d23349c1-0d41-41eb-a39a-5e7c1f034b5a" (UID: "d23349c1-0d41-41eb-a39a-5e7c1f034b5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.539643 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.539624 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-data" pod "d23349c1-0d41-41eb-a39a-5e7c1f034b5a" (UID: "d23349c1-0d41-41eb-a39a-5e7c1f034b5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.543681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.543656 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-podinfo" pod "7dcdd0a3-0820-4007-977e-c0595b979cb7" (UID: "7dcdd0a3-0820-4007-977e-c0595b979cb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:29.543843 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.543660 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-socket" pod "58e331e1-a05d-491b-a48c-913b3bb8f45e" (UID: "58e331e1-a05d-491b-a48c-913b3bb8f45e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.543937 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.543884 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-credential-socket" pod "7dcdd0a3-0820-4007-977e-c0595b979cb7" (UID: "7dcdd0a3-0820-4007-977e-c0595b979cb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.544113 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.544059 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-socket" pod "7dcdd0a3-0820-4007-977e-c0595b979cb7" (UID: "7dcdd0a3-0820-4007-977e-c0595b979cb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.544364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.544341 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-credential-socket" pod "d23349c1-0d41-41eb-a39a-5e7c1f034b5a" (UID: "d23349c1-0d41-41eb-a39a-5e7c1f034b5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.549680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.549654 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-podinfo" pod "d23349c1-0d41-41eb-a39a-5e7c1f034b5a" (UID: "d23349c1-0d41-41eb-a39a-5e7c1f034b5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:29.563263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.563226 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-envoy" pod "58e331e1-a05d-491b-a48c-913b3bb8f45e" (UID: "58e331e1-a05d-491b-a48c-913b3bb8f45e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.564322 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.564290 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-token" pod "d23349c1-0d41-41eb-a39a-5e7c1f034b5a" (UID: "d23349c1-0d41-41eb-a39a-5e7c1f034b5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.570824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.570793 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-kube-api-access-fhp4g" pod "58e331e1-a05d-491b-a48c-913b3bb8f45e" (UID: "58e331e1-a05d-491b-a48c-913b3bb8f45e"). InnerVolumeSpecName "kube-api-access-fhp4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.571326 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.571140 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-token" pod "7dcdd0a3-0820-4007-977e-c0595b979cb7" (UID: "7dcdd0a3-0820-4007-977e-c0595b979cb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.571326 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.571231 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-kube-api-access-hlddz" pod "d23349c1-0d41-41eb-a39a-5e7c1f034b5a" (UID: "d23349c1-0d41-41eb-a39a-5e7c1f034b5a"). InnerVolumeSpecName "kube-api-access-hlddz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.571326 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.571297 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-podinfo" pod "58e331e1-a05d-491b-a48c-913b3bb8f45e" (UID: "58e331e1-a05d-491b-a48c-913b3bb8f45e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:57:29.573097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.573042 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-token" pod "58e331e1-a05d-491b-a48c-913b3bb8f45e" (UID: "58e331e1-a05d-491b-a48c-913b3bb8f45e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:29.579192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.579132 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx"] Nov 02 17:57:29.583742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.583702 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-envoy" pod "d23349c1-0d41-41eb-a39a-5e7c1f034b5a" (UID: "d23349c1-0d41-41eb-a39a-5e7c1f034b5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:57:29.587070 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.587013 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-g2lrx"] Nov 02 17:57:29.640161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640125 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640161 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640159 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640170 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640179 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640188 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640198 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640207 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640219 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640233 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlddz\" (UniqueName: \"kubernetes.io/projected/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-kube-api-access-hlddz\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640246 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640256 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvcw7\" (UniqueName: \"kubernetes.io/projected/7dcdd0a3-0820-4007-977e-c0595b979cb7-kube-api-access-tvcw7\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640268 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640277 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640284 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640294 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhp4g\" (UniqueName: \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-kube-api-access-fhp4g\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640302 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640309 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e331e1-a05d-491b-a48c-913b3bb8f45e-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640317 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcdd0a3-0820-4007-977e-c0595b979cb7-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640325 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.640387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.640334 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23349c1-0d41-41eb-a39a-5e7c1f034b5a-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:29.768486 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:29.768450 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:30.239854 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.239818 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:30.453092 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.453043 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" Nov 02 17:57:30.453278 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.453057 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz" event={"ID":"d23349c1-0d41-41eb-a39a-5e7c1f034b5a","Type":"ContainerDied","Data":"2f919a0f2a1b96068ce79d45459545215fb8ae6e9e8ea129be0d0e40f6d1953b"} Nov 02 17:57:30.455291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.455260 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" event={"ID":"7dcdd0a3-0820-4007-977e-c0595b979cb7","Type":"ContainerDied","Data":"cba88597dec175a4b4cefb927e74a29386c996dc320952356f4ed69bc7ba9086"} Nov 02 17:57:30.455473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.455322 4310 scope.go:122] "RemoveContainer" containerID="740b0c0155ff13bdb5d7b597ca33fdb3baaa1933df177fdda2a156fcfc7e30dd" Nov 02 17:57:30.455473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.455274 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd" Nov 02 17:57:30.463586 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.463512 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" event={"ID":"58e331e1-a05d-491b-a48c-913b3bb8f45e","Type":"ContainerDied","Data":"10ee7b365f7722ad4238674de874adf741e137d97f70059c80601f992ac4d4c4"} Nov 02 17:57:30.463718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.463637 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng" Nov 02 17:57:30.469370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.469349 4310 scope.go:122] "RemoveContainer" containerID="14715dacc085e0427bb077b4ae06842eaefb155c908617b2ad4650f36506c2df" Nov 02 17:57:30.485424 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.485400 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz"] Nov 02 17:57:30.491742 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.491682 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-2x6sz"] Nov 02 17:57:30.528395 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.528350 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng"] Nov 02 17:57:30.535047 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.535016 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-cf7ng"] Nov 02 17:57:30.547744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.547712 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd"] Nov 02 17:57:30.555185 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.555156 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ea3867-openshift-default-7cfbcc5795-k2cdd"] Nov 02 17:57:30.588317 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.588287 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:30.710557 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.710519 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" path="/var/lib/kubelet/pods/58e331e1-a05d-491b-a48c-913b3bb8f45e/volumes" Nov 02 17:57:30.711435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.711410 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" path="/var/lib/kubelet/pods/7dcdd0a3-0820-4007-977e-c0595b979cb7/volumes" Nov 02 17:57:30.721674 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.712067 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" path="/var/lib/kubelet/pods/b2997e8d-e663-4e09-b1ea-a2a796d00aaa/volumes" Nov 02 17:57:30.722706 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:30.722677 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" path="/var/lib/kubelet/pods/d23349c1-0d41-41eb-a39a-5e7c1f034b5a/volumes" Nov 02 17:57:31.488256 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:31.488223 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:32.273756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:32.273725 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:32.994689 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:32.994656 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:33.522929 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:57:33.522895 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:57:33.682121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:33.682060 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:34.398763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:34.398731 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:35.029125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:35.029063 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv"] Nov 02 17:57:35.033896 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:35.033866 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fe38828-924qv"] Nov 02 17:57:35.054991 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:35.054958 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:35.476854 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:35.476819 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b"] Nov 02 17:57:35.541828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:35.541789 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:35.642779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:35.642749 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:36.068182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:36.068151 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:36.680632 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:36.680593 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d24d06a-1de2-46fe-a9a8-a3e63a60640b" path="/var/lib/kubelet/pods/6d24d06a-1de2-46fe-a9a8-a3e63a60640b/volumes" Nov 02 17:57:36.804699 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:36.804654 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:37.474573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.474524 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:37.685666 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.685635 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:37.834694 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.834617 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq"] Nov 02 17:57:37.835270 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835254 4310 cpu_manager.go:414] "containerMap: removing container" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" Nov 02 17:57:37.835338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835272 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" Nov 02 17:57:37.835338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835292 4310 cpu_manager.go:414] "containerMap: removing container" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" Nov 02 17:57:37.835338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835297 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" Nov 02 17:57:37.835338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835304 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a2d547b3-05f3-4108-a9df-272a923909f9" containerName="revoke-keys" Nov 02 17:57:37.835338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835310 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d547b3-05f3-4108-a9df-272a923909f9" containerName="revoke-keys" Nov 02 17:57:37.835338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835318 4310 cpu_manager.go:414] "containerMap: removing container" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" Nov 02 17:57:37.835338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835323 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" Nov 02 17:57:37.835338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835332 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" Nov 02 17:57:37.835338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835337 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" Nov 02 17:57:37.835627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835344 4310 cpu_manager.go:414] "containerMap: removing container" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" Nov 02 17:57:37.835627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835348 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" Nov 02 17:57:37.835627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835450 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" Nov 02 17:57:37.835627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835459 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2997e8d-e663-4e09-b1ea-a2a796d00aaa" containerName="istio-proxy" Nov 02 17:57:37.835627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835465 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d547b3-05f3-4108-a9df-272a923909f9" containerName="revoke-keys" Nov 02 17:57:37.835627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835472 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcdd0a3-0820-4007-977e-c0595b979cb7" containerName="istio-proxy" Nov 02 17:57:37.835627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835479 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d23349c1-0d41-41eb-a39a-5e7c1f034b5a" containerName="istio-proxy" Nov 02 17:57:37.835627 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.835485 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e331e1-a05d-491b-a48c-913b3bb8f45e" containerName="istio-proxy" Nov 02 17:57:37.841121 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.841102 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:37.850134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.848523 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:57:37.850134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.848613 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:57:37.850134 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.849293 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq"] Nov 02 17:57:37.893335 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.893301 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:37.893471 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.893350 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb79g\" (UniqueName: \"kubernetes.io/projected/ef1439af-171e-432e-af18-7cc9ae448e3f-kube-api-access-xb79g\") pod \"maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:37.893471 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.893437 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:37.994670 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.994621 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:37.994670 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.994678 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb79g\" (UniqueName: \"kubernetes.io/projected/ef1439af-171e-432e-af18-7cc9ae448e3f-kube-api-access-xb79g\") pod \"maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:37.994918 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.994716 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:37.995448 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.995426 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:37.997173 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:37.997152 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:38.005625 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:38.005603 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb79g\" (UniqueName: \"kubernetes.io/projected/ef1439af-171e-432e-af18-7cc9ae448e3f-kube-api-access-xb79g\") pod \"maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:38.170526 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:38.170446 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:38.453770 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:38.453739 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq"] Nov 02 17:57:38.454311 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:57:38.454277 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1439af_171e_432e_af18_7cc9ae448e3f.slice/crio-bb22e51e2ec83ab6ff75874e8c8c8deae026fbb896c5dc0155e95934f7c5793b WatchSource:0}: Error finding container bb22e51e2ec83ab6ff75874e8c8c8deae026fbb896c5dc0155e95934f7c5793b: Status 404 returned error can't find the container with id bb22e51e2ec83ab6ff75874e8c8c8deae026fbb896c5dc0155e95934f7c5793b Nov 02 17:57:38.516754 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:38.516699 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:38.612556 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:38.612502 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" event={"ID":"ef1439af-171e-432e-af18-7cc9ae448e3f","Type":"ContainerStarted","Data":"bb22e51e2ec83ab6ff75874e8c8c8deae026fbb896c5dc0155e95934f7c5793b"} Nov 02 17:57:39.207238 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.207160 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn"] Nov 02 17:57:39.212072 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.212046 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:39.225104 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.224434 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn"] Nov 02 17:57:39.312963 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.312925 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-924dc11d-shxxn\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:39.313164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.312976 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4675j\" (UniqueName: \"kubernetes.io/projected/7fd331b6-2583-4ea1-a326-3d18f31321c0-kube-api-access-4675j\") pod \"maas-api-revoke-sub-924dc11d-shxxn\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:39.313164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.313116 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-924dc11d-shxxn\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:39.413958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.413923 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-924dc11d-shxxn\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:39.414139 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.414107 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-924dc11d-shxxn\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:39.414207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.414143 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4675j\" (UniqueName: \"kubernetes.io/projected/7fd331b6-2583-4ea1-a326-3d18f31321c0-kube-api-access-4675j\") pod \"maas-api-revoke-sub-924dc11d-shxxn\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:39.414788 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.414760 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-924dc11d-shxxn\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:39.416432 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.416412 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-924dc11d-shxxn\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:39.424052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.424029 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4675j\" (UniqueName: \"kubernetes.io/projected/7fd331b6-2583-4ea1-a326-3d18f31321c0-kube-api-access-4675j\") pod \"maas-api-revoke-sub-924dc11d-shxxn\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:39.478650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.478627 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:39.554103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.551463 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:39.623765 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.623731 4310 generic.go:356] "Generic (PLEG): container finished" podID="ef1439af-171e-432e-af18-7cc9ae448e3f" containerID="3e4f6488223ab9f201f0d96d0993d108d9ac2a574ad818ee4b786c3a26f438a2" exitCode=0 Nov 02 17:57:39.623916 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.623812 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" event={"ID":"ef1439af-171e-432e-af18-7cc9ae448e3f","Type":"ContainerDied","Data":"3e4f6488223ab9f201f0d96d0993d108d9ac2a574ad818ee4b786c3a26f438a2"} Nov 02 17:57:39.865472 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:57:39.865446 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd331b6_2583_4ea1_a326_3d18f31321c0.slice/crio-0b9d89d79368469c032e7f1a353115587adceca3c3431bdd885d2aaea75d9f35 WatchSource:0}: Error finding container 0b9d89d79368469c032e7f1a353115587adceca3c3431bdd885d2aaea75d9f35: Status 404 returned error can't find the container with id 0b9d89d79368469c032e7f1a353115587adceca3c3431bdd885d2aaea75d9f35 Nov 02 17:57:39.867144 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:39.867123 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn"] Nov 02 17:57:40.029303 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.029256 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:40.157872 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.157834 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:40.253908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.253875 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:40.365990 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.365920 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:40.477688 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.477650 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" podUID="605b73ac-e668-4fc1-ad46-a901f791d92b" containerName="payload-processing" containerID="cri-o://cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad" gracePeriod=25 Nov 02 17:57:40.634159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.634124 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" event={"ID":"7fd331b6-2583-4ea1-a326-3d18f31321c0","Type":"ContainerStarted","Data":"2a26ae200905a2db98ac83a84eb9e540d15fb8c78b29c587d0095e4e21c6b77a"} Nov 02 17:57:40.634329 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.634178 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" event={"ID":"7fd331b6-2583-4ea1-a326-3d18f31321c0","Type":"ContainerStarted","Data":"0b9d89d79368469c032e7f1a353115587adceca3c3431bdd885d2aaea75d9f35"} Nov 02 17:57:40.653630 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.653578 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" podStartSLOduration=1.653564628 podStartE2EDuration="1.653564628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:40.650424855 +0000 UTC m=+1058.093507491" watchObservedRunningTime="2026-11-02 17:57:40.653564628 +0000 UTC m=+1058.096647264" Nov 02 17:57:40.839141 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.839115 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:40.868488 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.868456 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:40.950304 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.950277 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-xks-ca\") pod \"ef1439af-171e-432e-af18-7cc9ae448e3f\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " Nov 02 17:57:40.950509 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.950372 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-openshift-ca\") pod \"ef1439af-171e-432e-af18-7cc9ae448e3f\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " Nov 02 17:57:40.950509 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.950445 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1439af-171e-432e-af18-7cc9ae448e3f-kube-api-access-xb79g\" (UniqueName: \"kubernetes.io/projected/ef1439af-171e-432e-af18-7cc9ae448e3f-kube-api-access-xb79g\") pod \"ef1439af-171e-432e-af18-7cc9ae448e3f\" (UID: \"ef1439af-171e-432e-af18-7cc9ae448e3f\") " Nov 02 17:57:40.950713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.950683 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-openshift-ca" pod "ef1439af-171e-432e-af18-7cc9ae448e3f" (UID: "ef1439af-171e-432e-af18-7cc9ae448e3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:40.950799 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.950785 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:40.952562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.952536 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-xks-ca" pod "ef1439af-171e-432e-af18-7cc9ae448e3f" (UID: "ef1439af-171e-432e-af18-7cc9ae448e3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:40.952672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:40.952643 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1439af-171e-432e-af18-7cc9ae448e3f-kube-api-access-xb79g" pod "ef1439af-171e-432e-af18-7cc9ae448e3f" (UID: "ef1439af-171e-432e-af18-7cc9ae448e3f"). InnerVolumeSpecName "kube-api-access-xb79g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:41.039770 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:41.039688 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:41.052545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:41.052512 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb79g\" (UniqueName: \"kubernetes.io/projected/ef1439af-171e-432e-af18-7cc9ae448e3f-kube-api-access-xb79g\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:41.052545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:41.052539 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1439af-171e-432e-af18-7cc9ae448e3f-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:41.217302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:41.217261 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" podUID="605b73ac-e668-4fc1-ad46-a901f791d92b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:57:41.645610 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:41.645573 4310 generic.go:356] "Generic (PLEG): container finished" podID="7fd331b6-2583-4ea1-a326-3d18f31321c0" containerID="2a26ae200905a2db98ac83a84eb9e540d15fb8c78b29c587d0095e4e21c6b77a" exitCode=0 Nov 02 17:57:41.646112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:41.645702 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" event={"ID":"7fd331b6-2583-4ea1-a326-3d18f31321c0","Type":"ContainerDied","Data":"2a26ae200905a2db98ac83a84eb9e540d15fb8c78b29c587d0095e4e21c6b77a"} Nov 02 17:57:41.647542 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:41.647517 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" Nov 02 17:57:41.647686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:41.647554 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq" event={"ID":"ef1439af-171e-432e-af18-7cc9ae448e3f","Type":"ContainerDied","Data":"bb22e51e2ec83ab6ff75874e8c8c8deae026fbb896c5dc0155e95934f7c5793b"} Nov 02 17:57:41.647686 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:41.647600 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb22e51e2ec83ab6ff75874e8c8c8deae026fbb896c5dc0155e95934f7c5793b" Nov 02 17:57:41.764865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:41.764832 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:42.455396 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.455340 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:42.770357 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.770324 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:42.841415 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.841390 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:42.879481 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.879450 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-xks-ca\") pod \"7fd331b6-2583-4ea1-a326-3d18f31321c0\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " Nov 02 17:57:42.879639 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.879505 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd331b6-2583-4ea1-a326-3d18f31321c0-kube-api-access-4675j\" (UniqueName: \"kubernetes.io/projected/7fd331b6-2583-4ea1-a326-3d18f31321c0-kube-api-access-4675j\") pod \"7fd331b6-2583-4ea1-a326-3d18f31321c0\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " Nov 02 17:57:42.879704 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.879689 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-openshift-ca\") pod \"7fd331b6-2583-4ea1-a326-3d18f31321c0\" (UID: \"7fd331b6-2583-4ea1-a326-3d18f31321c0\") " Nov 02 17:57:42.880130 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.880061 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-openshift-ca" pod "7fd331b6-2583-4ea1-a326-3d18f31321c0" (UID: "7fd331b6-2583-4ea1-a326-3d18f31321c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:42.880421 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.880397 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:42.881789 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.881762 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-xks-ca" pod "7fd331b6-2583-4ea1-a326-3d18f31321c0" (UID: "7fd331b6-2583-4ea1-a326-3d18f31321c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:42.881909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.881836 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd331b6-2583-4ea1-a326-3d18f31321c0-kube-api-access-4675j" pod "7fd331b6-2583-4ea1-a326-3d18f31321c0" (UID: "7fd331b6-2583-4ea1-a326-3d18f31321c0"). InnerVolumeSpecName "kube-api-access-4675j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:42.981948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.981873 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd331b6-2583-4ea1-a326-3d18f31321c0-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:42.981948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:42.981902 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4675j\" (UniqueName: \"kubernetes.io/projected/7fd331b6-2583-4ea1-a326-3d18f31321c0-kube-api-access-4675j\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:43.271737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:43.271652 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:43.667972 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:43.667886 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" Nov 02 17:57:43.667972 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:43.667919 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn" event={"ID":"7fd331b6-2583-4ea1-a326-3d18f31321c0","Type":"ContainerDied","Data":"0b9d89d79368469c032e7f1a353115587adceca3c3431bdd885d2aaea75d9f35"} Nov 02 17:57:43.667972 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:43.667967 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9d89d79368469c032e7f1a353115587adceca3c3431bdd885d2aaea75d9f35" Nov 02 17:57:44.097580 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:44.097546 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:44.796179 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:44.796138 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:45.599878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:45.599841 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:46.393401 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:46.393361 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:47.201024 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:47.200987 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:47.820393 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:47.820358 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:48.077225 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:48.077136 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:48.079328 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:48.079300 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q"] Nov 02 17:57:48.089767 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:48.089708 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg"] Nov 02 17:57:48.282651 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:48.282621 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:49.109919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:49.109887 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:50.108781 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:50.108739 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:51.088873 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.088840 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:51.216850 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.216813 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" podUID="605b73ac-e668-4fc1-ad46-a901f791d92b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:57:51.909568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.909535 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk"] Nov 02 17:57:51.910153 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.910138 4310 cpu_manager.go:414] "containerMap: removing container" podUID="7fd331b6-2583-4ea1-a326-3d18f31321c0" containerName="revoke-keys" Nov 02 17:57:51.910207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.910155 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd331b6-2583-4ea1-a326-3d18f31321c0" containerName="revoke-keys" Nov 02 17:57:51.910207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.910182 4310 cpu_manager.go:414] "containerMap: removing container" podUID="ef1439af-171e-432e-af18-7cc9ae448e3f" containerName="revoke-keys" Nov 02 17:57:51.910207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.910187 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1439af-171e-432e-af18-7cc9ae448e3f" containerName="revoke-keys" Nov 02 17:57:51.910306 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.910294 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd331b6-2583-4ea1-a326-3d18f31321c0" containerName="revoke-keys" Nov 02 17:57:51.910340 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.910307 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1439af-171e-432e-af18-7cc9ae448e3f" containerName="revoke-keys" Nov 02 17:57:51.913884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.913863 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:51.923663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.923635 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm"] Nov 02 17:57:51.925033 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.925007 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:57:51.926158 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.925209 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:57:51.928779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.928758 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:51.933367 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.933343 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk"] Nov 02 17:57:51.943921 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:51.943887 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm"] Nov 02 17:57:52.051664 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.051630 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9e41d10-cwsmm\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:52.051840 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.051677 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8wf\" (UniqueName: \"kubernetes.io/projected/82ef9be3-7c5b-4407-b20b-277501637575-kube-api-access-vj8wf\") pod \"maas-api-revoke-sub-bf201658-v6vrk\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:52.051840 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.051796 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvg7\" (UniqueName: \"kubernetes.io/projected/d9f79edd-d030-40b3-a593-c6ea421cef43-kube-api-access-hwvg7\") pod \"maas-api-revoke-sub-a9e41d10-cwsmm\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:52.051840 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.051834 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf201658-v6vrk\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:52.051950 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.051896 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf201658-v6vrk\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:52.051987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.051972 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9e41d10-cwsmm\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:52.153840 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.153807 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvg7\" (UniqueName: \"kubernetes.io/projected/d9f79edd-d030-40b3-a593-c6ea421cef43-kube-api-access-hwvg7\") pod \"maas-api-revoke-sub-a9e41d10-cwsmm\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:52.154016 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.153849 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf201658-v6vrk\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:52.154016 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.153889 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf201658-v6vrk\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:52.154016 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.153965 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9e41d10-cwsmm\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:52.154376 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.154350 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9e41d10-cwsmm\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:52.154447 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.154394 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8wf\" (UniqueName: \"kubernetes.io/projected/82ef9be3-7c5b-4407-b20b-277501637575-kube-api-access-vj8wf\") pod \"maas-api-revoke-sub-bf201658-v6vrk\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:52.154787 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.154762 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf201658-v6vrk\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:52.155154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.155131 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9e41d10-cwsmm\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:52.156544 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.156522 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf201658-v6vrk\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:52.157459 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.157433 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9e41d10-cwsmm\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:52.187149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.187068 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8wf\" (UniqueName: \"kubernetes.io/projected/82ef9be3-7c5b-4407-b20b-277501637575-kube-api-access-vj8wf\") pod \"maas-api-revoke-sub-bf201658-v6vrk\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:52.204286 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.204263 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvg7\" (UniqueName: \"kubernetes.io/projected/d9f79edd-d030-40b3-a593-c6ea421cef43-kube-api-access-hwvg7\") pod \"maas-api-revoke-sub-a9e41d10-cwsmm\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:52.236526 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.236507 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:52.261114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.256101 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:52.347479 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.347442 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:52.615319 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:57:52.615231 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f79edd_d030_40b3_a593_c6ea421cef43.slice/crio-96cf104fdda08b8452e9ef21b376da1393218c94a71adf27482554f975fa9a19 WatchSource:0}: Error finding container 96cf104fdda08b8452e9ef21b376da1393218c94a71adf27482554f975fa9a19: Status 404 returned error can't find the container with id 96cf104fdda08b8452e9ef21b376da1393218c94a71adf27482554f975fa9a19 Nov 02 17:57:52.616793 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.616771 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm"] Nov 02 17:57:52.618398 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:57:52.618317 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ef9be3_7c5b_4407_b20b_277501637575.slice/crio-d236a8be07cb4a9a156649074eb49c5c8b22ca5c8b785e1cd858adedfc025920 WatchSource:0}: Error finding container d236a8be07cb4a9a156649074eb49c5c8b22ca5c8b785e1cd858adedfc025920: Status 404 returned error can't find the container with id d236a8be07cb4a9a156649074eb49c5c8b22ca5c8b785e1cd858adedfc025920 Nov 02 17:57:52.623591 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.623561 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk"] Nov 02 17:57:52.852352 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.852321 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" event={"ID":"82ef9be3-7c5b-4407-b20b-277501637575","Type":"ContainerStarted","Data":"d236a8be07cb4a9a156649074eb49c5c8b22ca5c8b785e1cd858adedfc025920"} Nov 02 17:57:52.853820 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:52.853797 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" event={"ID":"d9f79edd-d030-40b3-a593-c6ea421cef43","Type":"ContainerStarted","Data":"96cf104fdda08b8452e9ef21b376da1393218c94a71adf27482554f975fa9a19"} Nov 02 17:57:53.080032 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:53.079997 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" podUID="af852c12-fc47-48e9-ac51-7418a37792d5" containerName="payload-processing" containerID="cri-o://99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926" gracePeriod=25 Nov 02 17:57:53.093449 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:53.093420 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" podUID="378c5f34-0627-4df2-8096-5bb1e582c650" containerName="payload-pre-processing" containerID="cri-o://5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919" gracePeriod=25 Nov 02 17:57:53.325220 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:53.325180 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:53.864680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:53.864585 4310 generic.go:356] "Generic (PLEG): container finished" podID="d9f79edd-d030-40b3-a593-c6ea421cef43" containerID="e7f86aa3d92cf05cbd7c7be69730c59814a2f396ba2777d917a5228608e2f384" exitCode=0 Nov 02 17:57:53.864849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:53.864670 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" event={"ID":"d9f79edd-d030-40b3-a593-c6ea421cef43","Type":"ContainerDied","Data":"e7f86aa3d92cf05cbd7c7be69730c59814a2f396ba2777d917a5228608e2f384"} Nov 02 17:57:53.866500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:53.866480 4310 generic.go:356] "Generic (PLEG): container finished" podID="82ef9be3-7c5b-4407-b20b-277501637575" containerID="0b64e8b19a55878b3f49b04b3e07f162207f89e79b368708489d4180552c461f" exitCode=0 Nov 02 17:57:53.866621 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:53.866536 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" event={"ID":"82ef9be3-7c5b-4407-b20b-277501637575","Type":"ContainerDied","Data":"0b64e8b19a55878b3f49b04b3e07f162207f89e79b368708489d4180552c461f"} Nov 02 17:57:54.194806 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:54.194721 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:54.890551 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:54.890521 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:55.132779 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.132708 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:55.160890 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.160868 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-openshift-ca\") pod \"82ef9be3-7c5b-4407-b20b-277501637575\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " Nov 02 17:57:55.161150 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.161131 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-xks-ca\") pod \"82ef9be3-7c5b-4407-b20b-277501637575\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " Nov 02 17:57:55.161263 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.161177 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ef9be3-7c5b-4407-b20b-277501637575-kube-api-access-vj8wf\" (UniqueName: \"kubernetes.io/projected/82ef9be3-7c5b-4407-b20b-277501637575-kube-api-access-vj8wf\") pod \"82ef9be3-7c5b-4407-b20b-277501637575\" (UID: \"82ef9be3-7c5b-4407-b20b-277501637575\") " Nov 02 17:57:55.161971 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.161942 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-openshift-ca" pod "82ef9be3-7c5b-4407-b20b-277501637575" (UID: "82ef9be3-7c5b-4407-b20b-277501637575"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:55.164211 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.164180 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ef9be3-7c5b-4407-b20b-277501637575-kube-api-access-vj8wf" pod "82ef9be3-7c5b-4407-b20b-277501637575" (UID: "82ef9be3-7c5b-4407-b20b-277501637575"). InnerVolumeSpecName "kube-api-access-vj8wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:55.164517 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.164497 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:55.164592 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.164525 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8wf\" (UniqueName: \"kubernetes.io/projected/82ef9be3-7c5b-4407-b20b-277501637575-kube-api-access-vj8wf\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:55.164671 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.164654 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-xks-ca" pod "82ef9be3-7c5b-4407-b20b-277501637575" (UID: "82ef9be3-7c5b-4407-b20b-277501637575"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:55.171809 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.171784 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:55.258305 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.258270 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" podUID="af852c12-fc47-48e9-ac51-7418a37792d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:57:55.258475 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.258270 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" podUID="378c5f34-0627-4df2-8096-5bb1e582c650" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:57:55.265119 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.265066 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-openshift-ca\") pod \"d9f79edd-d030-40b3-a593-c6ea421cef43\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " Nov 02 17:57:55.265234 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.265128 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-xks-ca\") pod \"d9f79edd-d030-40b3-a593-c6ea421cef43\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " Nov 02 17:57:55.265307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.265253 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f79edd-d030-40b3-a593-c6ea421cef43-kube-api-access-hwvg7\" (UniqueName: \"kubernetes.io/projected/d9f79edd-d030-40b3-a593-c6ea421cef43-kube-api-access-hwvg7\") pod \"d9f79edd-d030-40b3-a593-c6ea421cef43\" (UID: \"d9f79edd-d030-40b3-a593-c6ea421cef43\") " Nov 02 17:57:55.265473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.265449 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-openshift-ca" pod "d9f79edd-d030-40b3-a593-c6ea421cef43" (UID: "d9f79edd-d030-40b3-a593-c6ea421cef43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:55.265850 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.265830 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ef9be3-7c5b-4407-b20b-277501637575-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:55.265926 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.265858 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:55.267219 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.267197 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-xks-ca" pod "d9f79edd-d030-40b3-a593-c6ea421cef43" (UID: "d9f79edd-d030-40b3-a593-c6ea421cef43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:55.267287 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.267203 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f79edd-d030-40b3-a593-c6ea421cef43-kube-api-access-hwvg7" pod "d9f79edd-d030-40b3-a593-c6ea421cef43" (UID: "d9f79edd-d030-40b3-a593-c6ea421cef43"). InnerVolumeSpecName "kube-api-access-hwvg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:55.367514 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.367484 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f79edd-d030-40b3-a593-c6ea421cef43-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:55.367514 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.367511 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwvg7\" (UniqueName: \"kubernetes.io/projected/d9f79edd-d030-40b3-a593-c6ea421cef43-kube-api-access-hwvg7\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:57:55.548068 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.548025 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:55.727100 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.726548 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:55.880148 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.873921 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:55.880148 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.874008 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz"] Nov 02 17:57:55.985107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.973283 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" event={"ID":"d9f79edd-d030-40b3-a593-c6ea421cef43","Type":"ContainerDied","Data":"96cf104fdda08b8452e9ef21b376da1393218c94a71adf27482554f975fa9a19"} Nov 02 17:57:55.985107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.973347 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96cf104fdda08b8452e9ef21b376da1393218c94a71adf27482554f975fa9a19" Nov 02 17:57:55.985107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.973600 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm" Nov 02 17:57:55.985107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.978362 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:55.985107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.981716 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" event={"ID":"82ef9be3-7c5b-4407-b20b-277501637575","Type":"ContainerDied","Data":"d236a8be07cb4a9a156649074eb49c5c8b22ca5c8b785e1cd858adedfc025920"} Nov 02 17:57:55.985107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.981768 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d236a8be07cb4a9a156649074eb49c5c8b22ca5c8b785e1cd858adedfc025920" Nov 02 17:57:55.985107 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.981838 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk" Nov 02 17:57:56.003183 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:55.986663 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9dfa27-db6dz"] Nov 02 17:57:56.301408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:56.301378 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:56.675884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:56.675807 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d547b3-05f3-4108-a9df-272a923909f9" path="/var/lib/kubelet/pods/a2d547b3-05f3-4108-a9df-272a923909f9/volumes" Nov 02 17:57:57.091341 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:57.091056 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:57.979556 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:57.979518 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:58.037308 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:58.037264 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs"] Nov 02 17:57:58.044047 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:58.044015 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0c60ca9-nwrfs"] Nov 02 17:57:58.674620 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:58.674589 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051ab575-90bd-46d9-9123-9ba74707d4b7" path="/var/lib/kubelet/pods/051ab575-90bd-46d9-9123-9ba74707d4b7/volumes" Nov 02 17:57:58.775231 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:58.775201 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:57:59.458534 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:57:59.458484 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:00.161717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:00.161680 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:00.774271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:00.774238 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:58:00.911957 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:00.911927 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605b73ac-e668-4fc1-ad46-a901f791d92b-kube-api-access-g95vd\" (UniqueName: \"kubernetes.io/projected/605b73ac-e668-4fc1-ad46-a901f791d92b-kube-api-access-g95vd\") pod \"605b73ac-e668-4fc1-ad46-a901f791d92b\" (UID: \"605b73ac-e668-4fc1-ad46-a901f791d92b\") " Nov 02 17:58:00.912149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:00.912033 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605b73ac-e668-4fc1-ad46-a901f791d92b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605b73ac-e668-4fc1-ad46-a901f791d92b-plugins-config-volume\") pod \"605b73ac-e668-4fc1-ad46-a901f791d92b\" (UID: \"605b73ac-e668-4fc1-ad46-a901f791d92b\") " Nov 02 17:58:00.912425 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:00.912395 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605b73ac-e668-4fc1-ad46-a901f791d92b-plugins-config-volume" pod "605b73ac-e668-4fc1-ad46-a901f791d92b" (UID: "605b73ac-e668-4fc1-ad46-a901f791d92b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:00.912633 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:00.912619 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605b73ac-e668-4fc1-ad46-a901f791d92b-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:00.914014 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:00.913993 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605b73ac-e668-4fc1-ad46-a901f791d92b-kube-api-access-g95vd" pod "605b73ac-e668-4fc1-ad46-a901f791d92b" (UID: "605b73ac-e668-4fc1-ad46-a901f791d92b"). InnerVolumeSpecName "kube-api-access-g95vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:00.980225 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:00.980186 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:01.016147 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:01.014247 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g95vd\" (UniqueName: \"kubernetes.io/projected/605b73ac-e668-4fc1-ad46-a901f791d92b-kube-api-access-g95vd\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:01.102251 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:01.102207 4310 generic.go:356] "Generic (PLEG): container finished" podID="605b73ac-e668-4fc1-ad46-a901f791d92b" containerID="cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad" exitCode=0 Nov 02 17:58:01.102558 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:01.102531 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" event={"ID":"605b73ac-e668-4fc1-ad46-a901f791d92b","Type":"ContainerDied","Data":"cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad"} Nov 02 17:58:01.111351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:01.105304 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" Nov 02 17:58:01.111516 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:01.111416 4310 scope.go:122] "RemoveContainer" containerID="cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad" Nov 02 17:58:01.111640 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:01.111339 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b" event={"ID":"605b73ac-e668-4fc1-ad46-a901f791d92b","Type":"ContainerDied","Data":"10d0d3ad21eb7fff720713a00a8886407815d0aad2b37e5fe4f6c80dd8fb94bc"} Nov 02 17:58:01.125274 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:01.125253 4310 scope.go:122] "RemoveContainer" containerID="cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad" Nov 02 17:58:01.125554 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:58:01.125532 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad\": container with ID starting with cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad not found: ID does not exist" containerID="cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad" Nov 02 17:58:01.125644 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:01.125561 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad"} err="failed to get container status \"cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad\": rpc error: code = NotFound desc = could not find container \"cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad\": container with ID starting with cd9e6b768bbdc1a9abe5c89659a8b712937f2c5cb66167cb6883cdc0272277ad not found: ID does not exist" Nov 02 17:58:01.139929 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:01.139909 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b"] Nov 02 17:58:01.144641 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:01.144620 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9dfa27-698dbcc947-pbv8b"] Nov 02 17:58:01.660586 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:01.660551 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:02.379708 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:02.379674 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:02.796631 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:02.764059 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605b73ac-e668-4fc1-ad46-a901f791d92b" path="/var/lib/kubelet/pods/605b73ac-e668-4fc1-ad46-a901f791d92b/volumes" Nov 02 17:58:03.282984 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.269152 4310 scope.go:122] "RemoveContainer" containerID="b57773de42695ac73c95e591f83f2ea173a22f3600d650794563c869c45f92ae" Nov 02 17:58:03.366828 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.362321 4310 scope.go:122] "RemoveContainer" containerID="bf74db187461117a124b9e0af96e12f879ca3f736d3f439d0e8a85b0a248bfce" Nov 02 17:58:03.451064 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.451025 4310 scope.go:122] "RemoveContainer" containerID="64c178df3d753debe23fb7d8228960119f769d9699964df2c8f997e20ebbd8fc" Nov 02 17:58:03.463072 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.463054 4310 scope.go:122] "RemoveContainer" containerID="2c66aab87c04bbfc22b42767e2436e7181d080845b7690d06f38a101bd6e51a7" Nov 02 17:58:03.472881 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.472828 4310 scope.go:122] "RemoveContainer" containerID="c66cc3b04c55c138792c16d3a34795e7359e2b09b9ca5ef3f91baf799cef01e9" Nov 02 17:58:03.482895 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.482874 4310 scope.go:122] "RemoveContainer" containerID="da02560a361b8e5c0e4d4959f7979d56da676bd666ea6bc98b0ac9cb3f384b68" Nov 02 17:58:03.492756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.492738 4310 scope.go:122] "RemoveContainer" containerID="a7aa8b5a1125c0ff17e6575affb256865f2ffdadb5caed2fcc1d4c812f499beb" Nov 02 17:58:03.506345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.506310 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:03.507560 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.507539 4310 scope.go:122] "RemoveContainer" containerID="dc4c33f892eefaf2fe587b695b35854f0cda290763560d8aa334d2e0dcb4342a" Nov 02 17:58:03.522347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.521866 4310 scope.go:122] "RemoveContainer" containerID="e2411c4012ca2531fabc393b8d0f26aa2fa367f1e5af2740e7344c21cdc345aa" Nov 02 17:58:03.550169 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:58:03.547282 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:58:03.566106 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.564939 4310 scope.go:122] "RemoveContainer" containerID="d81b2d1481a6809245086a5c78a830f7520b990722c9b5fd09664b74b51c3942" Nov 02 17:58:03.607233 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.607178 4310 scope.go:122] "RemoveContainer" containerID="8a03c43841de91c5cd1edcfe0d063d54c5ec2fff684f3a3393ae5c5da4a5c40a" Nov 02 17:58:03.618258 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.618216 4310 scope.go:122] "RemoveContainer" containerID="82022934e8bc8293cb026539743ca42c789c9cfb6cb47ce4ce30176f67525c13" Nov 02 17:58:03.627752 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.627731 4310 scope.go:122] "RemoveContainer" containerID="523c570822321d2e6da5c30d91cb5171b809ab43fdbec6f3f27c7b89a5156ac2" Nov 02 17:58:03.637948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.637931 4310 scope.go:122] "RemoveContainer" containerID="d63de7d3f35389a46fcf2a4684ac1cdd631cc7e46fc91e2333005232a5d0b252" Nov 02 17:58:03.647760 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.647737 4310 scope.go:122] "RemoveContainer" containerID="7176bad6074bb303b5241d6c28871543a70e0c827ca029442b02aebfb41478bc" Nov 02 17:58:03.657124 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.657073 4310 scope.go:122] "RemoveContainer" containerID="fda5dd6ffee7d845402278e92b2ea8fa4348bde281144d091e9ea8f1c605586a" Nov 02 17:58:03.667129 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.667102 4310 scope.go:122] "RemoveContainer" containerID="397d887daf3fa5fa1c97d3bf65b5bae48be942b49a35dcc87e47d636b0b53d01" Nov 02 17:58:03.679494 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:03.679476 4310 scope.go:122] "RemoveContainer" containerID="356ba5e35aed923627526ddab7dcd226bce478e0bd4895153f324ef416b217e1" Nov 02 17:58:04.449043 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:04.449005 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:04.993431 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:04.993395 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:05.219575 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.219530 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:05.260212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.258625 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" podUID="af852c12-fc47-48e9-ac51-7418a37792d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:58:05.260212 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.258695 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" podUID="378c5f34-0627-4df2-8096-5bb1e582c650" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:58:05.341172 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.341138 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk"] Nov 02 17:58:05.341384 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.341364 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" containerID="cri-o://48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424" gracePeriod=30 Nov 02 17:58:05.370771 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.370732 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng"] Nov 02 17:58:05.371000 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.370968 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" containerID="cri-o://b0f5b83f1498ae55e97eb2d8e2011edd938ff1c1177e55378836b52fc7556f06" gracePeriod=30 Nov 02 17:58:05.405914 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.405880 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4"] Nov 02 17:58:05.406146 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.406116 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" containerID="cri-o://b9db4b5dc82ba36d6f7a85bfe64a0d183b3b75cc1021971db5699b96d2306d06" gracePeriod=30 Nov 02 17:58:05.465125 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.457773 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:05.510110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.508243 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg"] Nov 02 17:58:05.510110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.508504 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" containerID="cri-o://a63f05f51af7127c1bf44efb3a1ba75b49f13dff92b3e8d56e858af64a221702" gracePeriod=30 Nov 02 17:58:05.882192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.882098 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:05.948609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.947401 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq"] Nov 02 17:58:05.953105 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:05.953057 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2a295c-4vxcq"] Nov 02 17:58:06.088776 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:06.088599 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:06.202438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:06.202361 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:06.448685 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:06.448653 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:06.675443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:06.675406 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:06.685636 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:06.685607 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1439af-171e-432e-af18-7cc9ae448e3f" path="/var/lib/kubelet/pods/ef1439af-171e-432e-af18-7cc9ae448e3f/volumes" Nov 02 17:58:06.824744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:06.818422 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:07.005487 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:06.998136 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:07.188387 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:07.188356 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:07.952991 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:07.950013 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:09.120915 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:09.120885 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:10.022728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.022690 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:10.511008 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.510932 4310 patch_prober.go:29] interesting pod/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 17:58:10.512066 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.511663 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 17:58:10.661958 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.661439 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:10.813832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.813801 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:58:10.924919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.920587 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-socket\") pod \"557797b3-5dcf-4373-8a05-a8aa9fba451e\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " Nov 02 17:58:10.924919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.920718 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-kube-api-access-2ghsh\" (UniqueName: \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-kube-api-access-2ghsh\") pod \"557797b3-5dcf-4373-8a05-a8aa9fba451e\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " Nov 02 17:58:10.924919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.920773 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-credential-socket\") pod \"557797b3-5dcf-4373-8a05-a8aa9fba451e\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " Nov 02 17:58:10.924919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.920821 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/557797b3-5dcf-4373-8a05-a8aa9fba451e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557797b3-5dcf-4373-8a05-a8aa9fba451e-istiod-ca-cert\") pod \"557797b3-5dcf-4373-8a05-a8aa9fba451e\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " Nov 02 17:58:10.924919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.920864 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-podinfo\") pod \"557797b3-5dcf-4373-8a05-a8aa9fba451e\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " Nov 02 17:58:10.924919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.920920 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-certs\") pod \"557797b3-5dcf-4373-8a05-a8aa9fba451e\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " Nov 02 17:58:10.924919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.920955 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-data\") pod \"557797b3-5dcf-4373-8a05-a8aa9fba451e\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " Nov 02 17:58:10.924919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.920998 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-envoy\") pod \"557797b3-5dcf-4373-8a05-a8aa9fba451e\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " Nov 02 17:58:10.924919 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.921035 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-token\" (UniqueName: \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-token\") pod \"557797b3-5dcf-4373-8a05-a8aa9fba451e\" (UID: \"557797b3-5dcf-4373-8a05-a8aa9fba451e\") " Nov 02 17:58:10.996246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.996190 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-credential-socket" pod "557797b3-5dcf-4373-8a05-a8aa9fba451e" (UID: "557797b3-5dcf-4373-8a05-a8aa9fba451e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:10.996438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:10.996290 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-socket" pod "557797b3-5dcf-4373-8a05-a8aa9fba451e" (UID: "557797b3-5dcf-4373-8a05-a8aa9fba451e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.021275 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.021197 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-data" pod "557797b3-5dcf-4373-8a05-a8aa9fba451e" (UID: "557797b3-5dcf-4373-8a05-a8aa9fba451e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.053314 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.040947 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.053895 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.053872 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.064099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.054117 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.064621 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.053269 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-envoy" pod "557797b3-5dcf-4373-8a05-a8aa9fba451e" (UID: "557797b3-5dcf-4373-8a05-a8aa9fba451e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.064819 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.053801 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-certs" pod "557797b3-5dcf-4373-8a05-a8aa9fba451e" (UID: "557797b3-5dcf-4373-8a05-a8aa9fba451e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.076227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.076195 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557797b3-5dcf-4373-8a05-a8aa9fba451e-istiod-ca-cert" pod "557797b3-5dcf-4373-8a05-a8aa9fba451e" (UID: "557797b3-5dcf-4373-8a05-a8aa9fba451e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:11.179324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.166468 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557797b3-5dcf-4373-8a05-a8aa9fba451e-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.179324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.166512 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.179324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.166527 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.316838 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.315976 4310 generic.go:356] "Generic (PLEG): container finished" podID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerID="b0f5b83f1498ae55e97eb2d8e2011edd938ff1c1177e55378836b52fc7556f06" exitCode=0 Nov 02 17:58:11.316838 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.316144 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" event={"ID":"4106a2f4-2564-4e87-a040-0c62ecaf2fb8","Type":"ContainerDied","Data":"b0f5b83f1498ae55e97eb2d8e2011edd938ff1c1177e55378836b52fc7556f06"} Nov 02 17:58:11.316838 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.316209 4310 scope.go:122] "RemoveContainer" containerID="5c429c0210d998fc1eb71770f8d81b6e13ef9140951ad66c65965920bbaa3c4b" Nov 02 17:58:11.364244 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.343408 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" event={"ID":"be7f8807-ebff-4a43-bb70-a161a1df954b","Type":"ContainerDied","Data":"a63f05f51af7127c1bf44efb3a1ba75b49f13dff92b3e8d56e858af64a221702"} Nov 02 17:58:11.364922 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.364683 4310 generic.go:356] "Generic (PLEG): container finished" podID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerID="a63f05f51af7127c1bf44efb3a1ba75b49f13dff92b3e8d56e858af64a221702" exitCode=0 Nov 02 17:58:11.368831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.367928 4310 generic.go:356] "Generic (PLEG): container finished" podID="153da82d-3725-4598-b171-e8b6b4205bf3" containerID="b9db4b5dc82ba36d6f7a85bfe64a0d183b3b75cc1021971db5699b96d2306d06" exitCode=0 Nov 02 17:58:11.368831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.368017 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" event={"ID":"153da82d-3725-4598-b171-e8b6b4205bf3","Type":"ContainerDied","Data":"b9db4b5dc82ba36d6f7a85bfe64a0d183b3b75cc1021971db5699b96d2306d06"} Nov 02 17:58:11.372271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.371510 4310 generic.go:356] "Generic (PLEG): container finished" podID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerID="48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424" exitCode=0 Nov 02 17:58:11.372271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.371607 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" event={"ID":"557797b3-5dcf-4373-8a05-a8aa9fba451e","Type":"ContainerDied","Data":"48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424"} Nov 02 17:58:11.372271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.371655 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" event={"ID":"557797b3-5dcf-4373-8a05-a8aa9fba451e","Type":"ContainerDied","Data":"f8f9e0b7286b96202b2aa13bd59e9aee3931795928ff73ff85bdd38202aab822"} Nov 02 17:58:11.372271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.371733 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk" Nov 02 17:58:11.484558 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.484502 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-podinfo" pod "557797b3-5dcf-4373-8a05-a8aa9fba451e" (UID: "557797b3-5dcf-4373-8a05-a8aa9fba451e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:11.485064 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.485022 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-token" pod "557797b3-5dcf-4373-8a05-a8aa9fba451e" (UID: "557797b3-5dcf-4373-8a05-a8aa9fba451e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.511154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.496978 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-kube-api-access-2ghsh" pod "557797b3-5dcf-4373-8a05-a8aa9fba451e" (UID: "557797b3-5dcf-4373-8a05-a8aa9fba451e"). InnerVolumeSpecName "kube-api-access-2ghsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.586132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.527273 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.640210 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.586437 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.640210 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.544009 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:11.640210 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.534564 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:58:11.660983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.657923 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:58:11.685847 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.685824 4310 scope.go:122] "RemoveContainer" containerID="48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424" Nov 02 17:58:11.687001 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.686970 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-credential-socket\") pod \"153da82d-3725-4598-b171-e8b6b4205bf3\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " Nov 02 17:58:11.687293 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.687275 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-token\" (UniqueName: \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-token\") pod \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " Nov 02 17:58:11.687449 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.687436 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-podinfo\") pod \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " Nov 02 17:58:11.687588 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.687576 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-certs\") pod \"153da82d-3725-4598-b171-e8b6b4205bf3\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " Nov 02 17:58:11.687696 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.687683 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-socket\") pod \"153da82d-3725-4598-b171-e8b6b4205bf3\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " Nov 02 17:58:11.687797 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.687786 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/153da82d-3725-4598-b171-e8b6b4205bf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153da82d-3725-4598-b171-e8b6b4205bf3-istio-podinfo\") pod \"153da82d-3725-4598-b171-e8b6b4205bf3\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " Nov 02 17:58:11.687906 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.687893 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-data\") pod \"153da82d-3725-4598-b171-e8b6b4205bf3\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " Nov 02 17:58:11.688050 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.688033 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-socket\") pod \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " Nov 02 17:58:11.688201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.688184 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-certs\") pod \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " Nov 02 17:58:11.688342 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.688327 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-istio-token\" (UniqueName: \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-istio-token\") pod \"153da82d-3725-4598-b171-e8b6b4205bf3\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " Nov 02 17:58:11.688480 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.688464 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-data\") pod \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " Nov 02 17:58:11.688599 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.688584 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153da82d-3725-4598-b171-e8b6b4205bf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153da82d-3725-4598-b171-e8b6b4205bf3-istiod-ca-cert\") pod \"153da82d-3725-4598-b171-e8b6b4205bf3\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " Nov 02 17:58:11.688713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.688701 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-envoy\") pod \"153da82d-3725-4598-b171-e8b6b4205bf3\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " Nov 02 17:58:11.688819 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.688806 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istiod-ca-cert\") pod \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " Nov 02 17:58:11.688943 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.688930 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-envoy\") pod \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " Nov 02 17:58:11.689071 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.689058 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-kube-api-access-2lvqn\" (UniqueName: \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-kube-api-access-2lvqn\") pod \"153da82d-3725-4598-b171-e8b6b4205bf3\" (UID: \"153da82d-3725-4598-b171-e8b6b4205bf3\") " Nov 02 17:58:11.689197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.689184 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-credential-socket\") pod \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " Nov 02 17:58:11.689345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.689332 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-kube-api-access-7w447\" (UniqueName: \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-kube-api-access-7w447\") pod \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\" (UID: \"4106a2f4-2564-4e87-a040-0c62ecaf2fb8\") " Nov 02 17:58:11.690099 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.690055 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ghsh\" (UniqueName: \"kubernetes.io/projected/557797b3-5dcf-4373-8a05-a8aa9fba451e-kube-api-access-2ghsh\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.690756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.690734 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-certs" pod "153da82d-3725-4598-b171-e8b6b4205bf3" (UID: "153da82d-3725-4598-b171-e8b6b4205bf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.691103 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.691069 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-socket" pod "153da82d-3725-4598-b171-e8b6b4205bf3" (UID: "153da82d-3725-4598-b171-e8b6b4205bf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.719227 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.719007 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istiod-ca-cert" pod "4106a2f4-2564-4e87-a040-0c62ecaf2fb8" (UID: "4106a2f4-2564-4e87-a040-0c62ecaf2fb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:11.719478 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.719301 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-data" pod "4106a2f4-2564-4e87-a040-0c62ecaf2fb8" (UID: "4106a2f4-2564-4e87-a040-0c62ecaf2fb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.721787 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.721214 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-podinfo" pod "4106a2f4-2564-4e87-a040-0c62ecaf2fb8" (UID: "4106a2f4-2564-4e87-a040-0c62ecaf2fb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:11.721787 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.721450 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-credential-socket" pod "153da82d-3725-4598-b171-e8b6b4205bf3" (UID: "153da82d-3725-4598-b171-e8b6b4205bf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.724132 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.723790 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-token" pod "4106a2f4-2564-4e87-a040-0c62ecaf2fb8" (UID: "4106a2f4-2564-4e87-a040-0c62ecaf2fb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.724786 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.724316 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153da82d-3725-4598-b171-e8b6b4205bf3-istiod-ca-cert" pod "153da82d-3725-4598-b171-e8b6b4205bf3" (UID: "153da82d-3725-4598-b171-e8b6b4205bf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:11.724786 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.724675 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-envoy" pod "4106a2f4-2564-4e87-a040-0c62ecaf2fb8" (UID: "4106a2f4-2564-4e87-a040-0c62ecaf2fb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.731713 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.731689 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-credential-socket" pod "4106a2f4-2564-4e87-a040-0c62ecaf2fb8" (UID: "4106a2f4-2564-4e87-a040-0c62ecaf2fb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.732810 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.732790 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-socket" pod "4106a2f4-2564-4e87-a040-0c62ecaf2fb8" (UID: "4106a2f4-2564-4e87-a040-0c62ecaf2fb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.733127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.733107 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-data" pod "153da82d-3725-4598-b171-e8b6b4205bf3" (UID: "153da82d-3725-4598-b171-e8b6b4205bf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.733420 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.733402 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-certs" pod "4106a2f4-2564-4e87-a040-0c62ecaf2fb8" (UID: "4106a2f4-2564-4e87-a040-0c62ecaf2fb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.745393 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.745350 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/153da82d-3725-4598-b171-e8b6b4205bf3-istio-podinfo" pod "153da82d-3725-4598-b171-e8b6b4205bf3" (UID: "153da82d-3725-4598-b171-e8b6b4205bf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:11.768094 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.754543 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-kube-api-access-2lvqn" pod "153da82d-3725-4598-b171-e8b6b4205bf3" (UID: "153da82d-3725-4598-b171-e8b6b4205bf3"). InnerVolumeSpecName "kube-api-access-2lvqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.768094 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.755180 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-istio-token" pod "153da82d-3725-4598-b171-e8b6b4205bf3" (UID: "153da82d-3725-4598-b171-e8b6b4205bf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.768094 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.755255 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-kube-api-access-7w447" pod "4106a2f4-2564-4e87-a040-0c62ecaf2fb8" (UID: "4106a2f4-2564-4e87-a040-0c62ecaf2fb8"). InnerVolumeSpecName "kube-api-access-7w447". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.768094 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.760437 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-envoy" pod "153da82d-3725-4598-b171-e8b6b4205bf3" (UID: "153da82d-3725-4598-b171-e8b6b4205bf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.771807 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.771488 4310 scope.go:122] "RemoveContainer" containerID="48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424" Nov 02 17:58:11.772489 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:58:11.772009 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424\": container with ID starting with 48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424 not found: ID does not exist" containerID="48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424" Nov 02 17:58:11.772489 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.772043 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424"} err="failed to get container status \"48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424\": rpc error: code = NotFound desc = could not find container \"48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424\": container with ID starting with 48305aea1ef70868ae740e2396c59487dfc6949577a72acf90d4be04bfa2a424 not found: ID does not exist" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792162 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792192 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792208 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792221 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153da82d-3725-4598-b171-e8b6b4205bf3-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792234 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792247 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792259 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792272 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792286 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792298 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153da82d-3725-4598-b171-e8b6b4205bf3-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792312 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792325 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792337 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792351 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lvqn\" (UniqueName: \"kubernetes.io/projected/153da82d-3725-4598-b171-e8b6b4205bf3-kube-api-access-2lvqn\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792365 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792377 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w447\" (UniqueName: \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-kube-api-access-7w447\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792391 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153da82d-3725-4598-b171-e8b6b4205bf3-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.795983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.792403 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4106a2f4-2564-4e87-a040-0c62ecaf2fb8-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.810539 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.810449 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk"] Nov 02 17:58:11.820862 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.820815 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zckrk"] Nov 02 17:58:11.888551 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.888526 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:58:11.893401 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893375 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-socket\") pod \"be7f8807-ebff-4a43-bb70-a161a1df954b\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " Nov 02 17:58:11.893512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893419 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-certs\") pod \"be7f8807-ebff-4a43-bb70-a161a1df954b\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " Nov 02 17:58:11.893512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893486 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-kube-api-access-nr44h\" (UniqueName: \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-kube-api-access-nr44h\") pod \"be7f8807-ebff-4a43-bb70-a161a1df954b\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " Nov 02 17:58:11.893602 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893574 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-data\") pod \"be7f8807-ebff-4a43-bb70-a161a1df954b\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " Nov 02 17:58:11.893661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893602 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-socket" pod "be7f8807-ebff-4a43-bb70-a161a1df954b" (UID: "be7f8807-ebff-4a43-bb70-a161a1df954b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.893661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893638 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-token\" (UniqueName: \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-token\") pod \"be7f8807-ebff-4a43-bb70-a161a1df954b\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " Nov 02 17:58:11.893780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893678 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-podinfo\") pod \"be7f8807-ebff-4a43-bb70-a161a1df954b\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " Nov 02 17:58:11.893780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893709 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-credential-socket\") pod \"be7f8807-ebff-4a43-bb70-a161a1df954b\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " Nov 02 17:58:11.893780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893726 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-certs" pod "be7f8807-ebff-4a43-bb70-a161a1df954b" (UID: "be7f8807-ebff-4a43-bb70-a161a1df954b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.893940 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893855 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be7f8807-ebff-4a43-bb70-a161a1df954b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7f8807-ebff-4a43-bb70-a161a1df954b-istiod-ca-cert\") pod \"be7f8807-ebff-4a43-bb70-a161a1df954b\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " Nov 02 17:58:11.893940 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893864 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-data" pod "be7f8807-ebff-4a43-bb70-a161a1df954b" (UID: "be7f8807-ebff-4a43-bb70-a161a1df954b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.894051 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893952 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-envoy\") pod \"be7f8807-ebff-4a43-bb70-a161a1df954b\" (UID: \"be7f8807-ebff-4a43-bb70-a161a1df954b\") " Nov 02 17:58:11.894051 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.893966 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-credential-socket" pod "be7f8807-ebff-4a43-bb70-a161a1df954b" (UID: "be7f8807-ebff-4a43-bb70-a161a1df954b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.894266 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.894241 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7f8807-ebff-4a43-bb70-a161a1df954b-istiod-ca-cert" pod "be7f8807-ebff-4a43-bb70-a161a1df954b" (UID: "be7f8807-ebff-4a43-bb70-a161a1df954b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:11.894545 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.894524 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.894646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.894552 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7f8807-ebff-4a43-bb70-a161a1df954b-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.894646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.894569 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.894646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.894585 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.894646 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.894599 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.895801 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.895775 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-podinfo" pod "be7f8807-ebff-4a43-bb70-a161a1df954b" (UID: "be7f8807-ebff-4a43-bb70-a161a1df954b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:11.895925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.895805 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-kube-api-access-nr44h" pod "be7f8807-ebff-4a43-bb70-a161a1df954b" (UID: "be7f8807-ebff-4a43-bb70-a161a1df954b"). InnerVolumeSpecName "kube-api-access-nr44h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.895925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.895819 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-token" pod "be7f8807-ebff-4a43-bb70-a161a1df954b" (UID: "be7f8807-ebff-4a43-bb70-a161a1df954b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:11.895925 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.895881 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-envoy" pod "be7f8807-ebff-4a43-bb70-a161a1df954b" (UID: "be7f8807-ebff-4a43-bb70-a161a1df954b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:11.995890 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.995855 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr44h\" (UniqueName: \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-kube-api-access-nr44h\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.995890 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.995885 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.995890 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.995895 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:11.996178 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:11.995904 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7f8807-ebff-4a43-bb70-a161a1df954b-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:12.380783 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.380701 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" Nov 02 17:58:12.380783 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.380709 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4" event={"ID":"153da82d-3725-4598-b171-e8b6b4205bf3","Type":"ContainerDied","Data":"94b6e11cfe8779b46ac5abc5118c953de9b24ee468d08c99c96cbf827e287ded"} Nov 02 17:58:12.381007 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.380782 4310 scope.go:122] "RemoveContainer" containerID="b9db4b5dc82ba36d6f7a85bfe64a0d183b3b75cc1021971db5699b96d2306d06" Nov 02 17:58:12.383741 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.383714 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" Nov 02 17:58:12.383866 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.383789 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng" event={"ID":"4106a2f4-2564-4e87-a040-0c62ecaf2fb8","Type":"ContainerDied","Data":"55fb2ffb26612387fb8f6e7586b49b6d1ede5e780655a6250138806b49f839cc"} Nov 02 17:58:12.385408 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.385373 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" event={"ID":"be7f8807-ebff-4a43-bb70-a161a1df954b","Type":"ContainerDied","Data":"293400f2c8809c51f927cb83c20407cfa123735b92590140b0c7f3468a656c17"} Nov 02 17:58:12.385532 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.385434 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg" Nov 02 17:58:12.398159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.394983 4310 scope.go:122] "RemoveContainer" containerID="b0f5b83f1498ae55e97eb2d8e2011edd938ff1c1177e55378836b52fc7556f06" Nov 02 17:58:12.398159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.395223 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:12.416513 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.416490 4310 scope.go:122] "RemoveContainer" containerID="a63f05f51af7127c1bf44efb3a1ba75b49f13dff92b3e8d56e858af64a221702" Nov 02 17:58:12.445456 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.445383 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4"] Nov 02 17:58:12.488015 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.487978 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-ncwz4"] Nov 02 17:58:12.499758 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.499454 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng"] Nov 02 17:58:12.511008 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.508726 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-8h7ng"] Nov 02 17:58:12.529536 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.529500 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg"] Nov 02 17:58:12.536631 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.536600 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9dfa27-openshift-default-58b97f94c7-zr9cg"] Nov 02 17:58:12.675404 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.675326 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" path="/var/lib/kubelet/pods/153da82d-3725-4598-b171-e8b6b4205bf3/volumes" Nov 02 17:58:12.675996 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.675981 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" path="/var/lib/kubelet/pods/4106a2f4-2564-4e87-a040-0c62ecaf2fb8/volumes" Nov 02 17:58:12.676590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.676576 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" path="/var/lib/kubelet/pods/557797b3-5dcf-4373-8a05-a8aa9fba451e/volumes" Nov 02 17:58:12.677591 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:12.677573 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" path="/var/lib/kubelet/pods/be7f8807-ebff-4a43-bb70-a161a1df954b/volumes" Nov 02 17:58:13.046159 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.044263 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv"] Nov 02 17:58:13.049201 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.049171 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611811fe-8bhkv"] Nov 02 17:58:13.367934 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.367906 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:58:13.391892 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.391861 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:58:13.401540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.401512 4310 generic.go:356] "Generic (PLEG): container finished" podID="af852c12-fc47-48e9-ac51-7418a37792d5" containerID="99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926" exitCode=0 Nov 02 17:58:13.401728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.401570 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" Nov 02 17:58:13.401728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.401603 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" event={"ID":"af852c12-fc47-48e9-ac51-7418a37792d5","Type":"ContainerDied","Data":"99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926"} Nov 02 17:58:13.401728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.401654 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q" event={"ID":"af852c12-fc47-48e9-ac51-7418a37792d5","Type":"ContainerDied","Data":"dc53c1e5ca9201edff4e7acb14fb502dfc5dfdd9fa2f2b7ba84c8059d8afc70b"} Nov 02 17:58:13.401728 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.401685 4310 scope.go:122] "RemoveContainer" containerID="99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926" Nov 02 17:58:13.406150 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.406121 4310 generic.go:356] "Generic (PLEG): container finished" podID="378c5f34-0627-4df2-8096-5bb1e582c650" containerID="5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919" exitCode=0 Nov 02 17:58:13.406292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.406173 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" Nov 02 17:58:13.406292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.406191 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" event={"ID":"378c5f34-0627-4df2-8096-5bb1e582c650","Type":"ContainerDied","Data":"5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919"} Nov 02 17:58:13.406292 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.406276 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg" event={"ID":"378c5f34-0627-4df2-8096-5bb1e582c650","Type":"ContainerDied","Data":"fe7fb37954d23b96881a49fdff49ad3099464df1dfceb81f402844f45fa78293"} Nov 02 17:58:13.410293 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.410251 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:13.413711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.413653 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378c5f34-0627-4df2-8096-5bb1e582c650-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378c5f34-0627-4df2-8096-5bb1e582c650-plugins-config-volume\") pod \"378c5f34-0627-4df2-8096-5bb1e582c650\" (UID: \"378c5f34-0627-4df2-8096-5bb1e582c650\") " Nov 02 17:58:13.413846 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.413776 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af852c12-fc47-48e9-ac51-7418a37792d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af852c12-fc47-48e9-ac51-7418a37792d5-plugins-config-volume\") pod \"af852c12-fc47-48e9-ac51-7418a37792d5\" (UID: \"af852c12-fc47-48e9-ac51-7418a37792d5\") " Nov 02 17:58:13.413970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.413895 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378c5f34-0627-4df2-8096-5bb1e582c650-kube-api-access-z9pj5\" (UniqueName: \"kubernetes.io/projected/378c5f34-0627-4df2-8096-5bb1e582c650-kube-api-access-z9pj5\") pod \"378c5f34-0627-4df2-8096-5bb1e582c650\" (UID: \"378c5f34-0627-4df2-8096-5bb1e582c650\") " Nov 02 17:58:13.413970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.413936 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af852c12-fc47-48e9-ac51-7418a37792d5-kube-api-access-vhgvw\" (UniqueName: \"kubernetes.io/projected/af852c12-fc47-48e9-ac51-7418a37792d5-kube-api-access-vhgvw\") pod \"af852c12-fc47-48e9-ac51-7418a37792d5\" (UID: \"af852c12-fc47-48e9-ac51-7418a37792d5\") " Nov 02 17:58:13.415271 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.415235 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af852c12-fc47-48e9-ac51-7418a37792d5-plugins-config-volume" pod "af852c12-fc47-48e9-ac51-7418a37792d5" (UID: "af852c12-fc47-48e9-ac51-7418a37792d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:13.415700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.415680 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378c5f34-0627-4df2-8096-5bb1e582c650-plugins-config-volume" pod "378c5f34-0627-4df2-8096-5bb1e582c650" (UID: "378c5f34-0627-4df2-8096-5bb1e582c650"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:13.418015 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.417987 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378c5f34-0627-4df2-8096-5bb1e582c650-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:13.418151 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.418020 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af852c12-fc47-48e9-ac51-7418a37792d5-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:13.419546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.419507 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af852c12-fc47-48e9-ac51-7418a37792d5-kube-api-access-vhgvw" pod "af852c12-fc47-48e9-ac51-7418a37792d5" (UID: "af852c12-fc47-48e9-ac51-7418a37792d5"). InnerVolumeSpecName "kube-api-access-vhgvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:13.428935 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.428903 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378c5f34-0627-4df2-8096-5bb1e582c650-kube-api-access-z9pj5" pod "378c5f34-0627-4df2-8096-5bb1e582c650" (UID: "378c5f34-0627-4df2-8096-5bb1e582c650"). InnerVolumeSpecName "kube-api-access-z9pj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:13.504721 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.504689 4310 scope.go:122] "RemoveContainer" containerID="99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926" Nov 02 17:58:13.505140 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:58:13.505067 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926\": container with ID starting with 99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926 not found: ID does not exist" containerID="99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926" Nov 02 17:58:13.505275 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.505137 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926"} err="failed to get container status \"99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926\": rpc error: code = NotFound desc = could not find container \"99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926\": container with ID starting with 99fbb326bc7e7cd4bd45ffb98b750d698460e4be233dba292bda73eb6a7ff926 not found: ID does not exist" Nov 02 17:58:13.505275 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.505165 4310 scope.go:122] "RemoveContainer" containerID="5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919" Nov 02 17:58:13.524558 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.524216 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9pj5\" (UniqueName: \"kubernetes.io/projected/378c5f34-0627-4df2-8096-5bb1e582c650-kube-api-access-z9pj5\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:13.524558 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.524259 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhgvw\" (UniqueName: \"kubernetes.io/projected/af852c12-fc47-48e9-ac51-7418a37792d5-kube-api-access-vhgvw\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:13.526757 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.525791 4310 scope.go:122] "RemoveContainer" containerID="5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919" Nov 02 17:58:13.526757 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:58:13.526272 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919\": container with ID starting with 5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919 not found: ID does not exist" containerID="5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919" Nov 02 17:58:13.526757 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.526308 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919"} err="failed to get container status \"5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919\": rpc error: code = NotFound desc = could not find container \"5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919\": container with ID starting with 5ebd585fb63c4d3090b401a8353157bb42115636b6cd8a28e316b3e37f0e6919 not found: ID does not exist" Nov 02 17:58:13.730176 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.730149 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q"] Nov 02 17:58:13.736578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.736550 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2a295c-59dfd8fb57-ghw4q"] Nov 02 17:58:13.753878 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.753851 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg"] Nov 02 17:58:13.758402 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:13.758380 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2a295c-6965b9c7bc-zvcvg"] Nov 02 17:58:14.065987 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.065949 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:14.181836 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.181795 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:14.559041 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.559005 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w"] Nov 02 17:58:14.559250 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.559214 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" containerID="cri-o://2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53" gracePeriod=30 Nov 02 17:58:14.635330 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.633809 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74"] Nov 02 17:58:14.635330 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.634056 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerName="istio-proxy" containerID="cri-o://40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0" gracePeriod=30 Nov 02 17:58:14.636797 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.636735 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:14.679177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.679137 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2"] Nov 02 17:58:14.722097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.720122 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" containerID="cri-o://5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6" gracePeriod=30 Nov 02 17:58:14.756535 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.756504 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378c5f34-0627-4df2-8096-5bb1e582c650" path="/var/lib/kubelet/pods/378c5f34-0627-4df2-8096-5bb1e582c650/volumes" Nov 02 17:58:14.759046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.759018 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6" path="/var/lib/kubelet/pods/4fc92f38-7bd6-45b6-999b-5bb7a84d1ec6/volumes" Nov 02 17:58:14.760799 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.760778 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af852c12-fc47-48e9-ac51-7418a37792d5" path="/var/lib/kubelet/pods/af852c12-fc47-48e9-ac51-7418a37792d5/volumes" Nov 02 17:58:14.761810 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.761792 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc"] Nov 02 17:58:14.762507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.762481 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" podUID="aed79116-9b99-4427-814f-e8af212f5126" containerName="istio-proxy" containerID="cri-o://8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4" gracePeriod=30 Nov 02 17:58:14.814998 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.814913 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:14.966737 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:14.966702 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:15.398656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:15.398620 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:16.175072 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:16.175032 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:16.902237 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:16.902200 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:17.812040 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:17.812010 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:18.510468 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:18.510434 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:18.618376 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:18.618342 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:18.719250 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:18.719213 4310 patch_prober.go:29] interesting pod/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:58:18.719403 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:18.719272 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:58:19.423848 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:19.423815 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:19.898056 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:19.896823 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:58:19.950591 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:19.950162 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:58:20.035215 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035184 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-token\" (UniqueName: \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-token\") pod \"38d15b4f-697d-40ac-9299-fbf4b413127c\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " Nov 02 17:58:20.035386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035258 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-socket\") pod \"38d15b4f-697d-40ac-9299-fbf4b413127c\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " Nov 02 17:58:20.035386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035296 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-certs\") pod \"38d15b4f-697d-40ac-9299-fbf4b413127c\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " Nov 02 17:58:20.035386 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035354 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-credential-socket\") pod \"0253d943-e554-4580-b042-cabae8436768\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " Nov 02 17:58:20.035562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035389 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-istio-token\" (UniqueName: \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-istio-token\") pod \"0253d943-e554-4580-b042-cabae8436768\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " Nov 02 17:58:20.035562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035417 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-kube-api-access-jnk78\" (UniqueName: \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-kube-api-access-jnk78\") pod \"0253d943-e554-4580-b042-cabae8436768\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " Nov 02 17:58:20.035562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035460 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-certs\") pod \"0253d943-e554-4580-b042-cabae8436768\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " Nov 02 17:58:20.035562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035491 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-data\") pod \"38d15b4f-697d-40ac-9299-fbf4b413127c\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " Nov 02 17:58:20.035562 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035537 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-socket\") pod \"0253d943-e554-4580-b042-cabae8436768\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " Nov 02 17:58:20.035849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035572 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-kube-api-access-zbq9v\" (UniqueName: \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-kube-api-access-zbq9v\") pod \"38d15b4f-697d-40ac-9299-fbf4b413127c\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " Nov 02 17:58:20.035849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035599 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-envoy\") pod \"38d15b4f-697d-40ac-9299-fbf4b413127c\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " Nov 02 17:58:20.035849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035657 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0253d943-e554-4580-b042-cabae8436768-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0253d943-e554-4580-b042-cabae8436768-istio-podinfo\") pod \"0253d943-e554-4580-b042-cabae8436768\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " Nov 02 17:58:20.035849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035715 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0253d943-e554-4580-b042-cabae8436768-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0253d943-e554-4580-b042-cabae8436768-istiod-ca-cert\") pod \"0253d943-e554-4580-b042-cabae8436768\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " Nov 02 17:58:20.035849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035743 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-envoy\") pod \"0253d943-e554-4580-b042-cabae8436768\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " Nov 02 17:58:20.035849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035795 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-credential-socket\") pod \"38d15b4f-697d-40ac-9299-fbf4b413127c\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " Nov 02 17:58:20.035849 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035846 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d15b4f-697d-40ac-9299-fbf4b413127c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d15b4f-697d-40ac-9299-fbf4b413127c-istiod-ca-cert\") pod \"38d15b4f-697d-40ac-9299-fbf4b413127c\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " Nov 02 17:58:20.036259 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035945 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-podinfo\") pod \"38d15b4f-697d-40ac-9299-fbf4b413127c\" (UID: \"38d15b4f-697d-40ac-9299-fbf4b413127c\") " Nov 02 17:58:20.036259 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.035989 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-data\") pod \"0253d943-e554-4580-b042-cabae8436768\" (UID: \"0253d943-e554-4580-b042-cabae8436768\") " Nov 02 17:58:20.036989 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.036581 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-socket" pod "0253d943-e554-4580-b042-cabae8436768" (UID: "0253d943-e554-4580-b042-cabae8436768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.036989 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.036793 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-data" pod "0253d943-e554-4580-b042-cabae8436768" (UID: "0253d943-e554-4580-b042-cabae8436768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.040164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.039239 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0253d943-e554-4580-b042-cabae8436768-istiod-ca-cert" pod "0253d943-e554-4580-b042-cabae8436768" (UID: "0253d943-e554-4580-b042-cabae8436768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:20.040164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.039931 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-socket" pod "38d15b4f-697d-40ac-9299-fbf4b413127c" (UID: "38d15b4f-697d-40ac-9299-fbf4b413127c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.040164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.039943 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d15b4f-697d-40ac-9299-fbf4b413127c-istiod-ca-cert" pod "38d15b4f-697d-40ac-9299-fbf4b413127c" (UID: "38d15b4f-697d-40ac-9299-fbf4b413127c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:20.040164 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.040142 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-certs" pod "38d15b4f-697d-40ac-9299-fbf4b413127c" (UID: "38d15b4f-697d-40ac-9299-fbf4b413127c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.040437 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.040331 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-credential-socket" pod "0253d943-e554-4580-b042-cabae8436768" (UID: "0253d943-e554-4580-b042-cabae8436768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.040543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.040046 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-token" pod "38d15b4f-697d-40ac-9299-fbf4b413127c" (UID: "38d15b4f-697d-40ac-9299-fbf4b413127c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.041717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.041326 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-credential-socket" pod "38d15b4f-697d-40ac-9299-fbf4b413127c" (UID: "38d15b4f-697d-40ac-9299-fbf4b413127c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.042806 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.042468 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0253d943-e554-4580-b042-cabae8436768-istio-podinfo" pod "0253d943-e554-4580-b042-cabae8436768" (UID: "0253d943-e554-4580-b042-cabae8436768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:20.043324 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.043005 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-data" pod "38d15b4f-697d-40ac-9299-fbf4b413127c" (UID: "38d15b4f-697d-40ac-9299-fbf4b413127c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.043661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.043496 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-istio-token" pod "0253d943-e554-4580-b042-cabae8436768" (UID: "0253d943-e554-4580-b042-cabae8436768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.045154 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.045130 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-kube-api-access-jnk78" pod "0253d943-e554-4580-b042-cabae8436768" (UID: "0253d943-e554-4580-b042-cabae8436768"). InnerVolumeSpecName "kube-api-access-jnk78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.045268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.045242 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-envoy" pod "38d15b4f-697d-40ac-9299-fbf4b413127c" (UID: "38d15b4f-697d-40ac-9299-fbf4b413127c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.045842 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.045805 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-podinfo" pod "38d15b4f-697d-40ac-9299-fbf4b413127c" (UID: "38d15b4f-697d-40ac-9299-fbf4b413127c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:20.049254 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.049224 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:58:20.050529 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.050506 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-kube-api-access-zbq9v" pod "38d15b4f-697d-40ac-9299-fbf4b413127c" (UID: "38d15b4f-697d-40ac-9299-fbf4b413127c"). InnerVolumeSpecName "kube-api-access-zbq9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.052364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.052338 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-envoy" pod "0253d943-e554-4580-b042-cabae8436768" (UID: "0253d943-e554-4580-b042-cabae8436768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.056923 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.056885 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-certs" pod "0253d943-e554-4580-b042-cabae8436768" (UID: "0253d943-e554-4580-b042-cabae8436768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.099008 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.098974 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:58:20.137340 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.137302 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istiod-ca-cert\") pod \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " Nov 02 17:58:20.137528 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.137398 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-token\" (UniqueName: \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-token\") pod \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " Nov 02 17:58:20.137528 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.137467 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-certs\") pod \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " Nov 02 17:58:20.137528 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.137514 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-envoy\") pod \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " Nov 02 17:58:20.137711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.137541 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-socket\") pod \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " Nov 02 17:58:20.137711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.137577 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-credential-socket\") pod \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " Nov 02 17:58:20.137711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.137645 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-podinfo\") pod \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " Nov 02 17:58:20.137711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.137698 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-kube-api-access-4xk2n\" (UniqueName: \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-kube-api-access-4xk2n\") pod \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " Nov 02 17:58:20.137924 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.137744 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-data\") pod \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\" (UID: \"6069e34a-83a9-4812-85d4-ef0fd0e5173d\") " Nov 02 17:58:20.137924 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.137749 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istiod-ca-cert" pod "6069e34a-83a9-4812-85d4-ef0fd0e5173d" (UID: "6069e34a-83a9-4812-85d4-ef0fd0e5173d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:20.138216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138190 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-socket" pod "6069e34a-83a9-4812-85d4-ef0fd0e5173d" (UID: "6069e34a-83a9-4812-85d4-ef0fd0e5173d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138408 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-data" pod "6069e34a-83a9-4812-85d4-ef0fd0e5173d" (UID: "6069e34a-83a9-4812-85d4-ef0fd0e5173d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138335 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138535 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138552 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138570 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnk78\" (UniqueName: \"kubernetes.io/projected/0253d943-e554-4580-b042-cabae8436768-kube-api-access-jnk78\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138584 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138598 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138613 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138627 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbq9v\" (UniqueName: \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-kube-api-access-zbq9v\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138642 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138656 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0253d943-e554-4580-b042-cabae8436768-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.138667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138669 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0253d943-e554-4580-b042-cabae8436768-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.139428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138682 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.139428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138698 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.139428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138712 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.139428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138725 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d15b4f-697d-40ac-9299-fbf4b413127c-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.139428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138739 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.139428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138752 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0253d943-e554-4580-b042-cabae8436768-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.139428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138767 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d15b4f-697d-40ac-9299-fbf4b413127c-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.139428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138780 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.139428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138794 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d15b4f-697d-40ac-9299-fbf4b413127c-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.139428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.138853 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-credential-socket" pod "6069e34a-83a9-4812-85d4-ef0fd0e5173d" (UID: "6069e34a-83a9-4812-85d4-ef0fd0e5173d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.139428 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.139021 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-certs" pod "6069e34a-83a9-4812-85d4-ef0fd0e5173d" (UID: "6069e34a-83a9-4812-85d4-ef0fd0e5173d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.140137 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.140111 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-token" pod "6069e34a-83a9-4812-85d4-ef0fd0e5173d" (UID: "6069e34a-83a9-4812-85d4-ef0fd0e5173d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.140389 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.140366 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-kube-api-access-4xk2n" pod "6069e34a-83a9-4812-85d4-ef0fd0e5173d" (UID: "6069e34a-83a9-4812-85d4-ef0fd0e5173d"). InnerVolumeSpecName "kube-api-access-4xk2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.140597 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.140571 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-podinfo" pod "6069e34a-83a9-4812-85d4-ef0fd0e5173d" (UID: "6069e34a-83a9-4812-85d4-ef0fd0e5173d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:20.141492 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.141472 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-envoy" pod "6069e34a-83a9-4812-85d4-ef0fd0e5173d" (UID: "6069e34a-83a9-4812-85d4-ef0fd0e5173d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.239230 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.239199 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-credential-socket\") pod \"aed79116-9b99-4427-814f-e8af212f5126\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " Nov 02 17:58:20.239406 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.239239 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-envoy\") pod \"aed79116-9b99-4427-814f-e8af212f5126\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " Nov 02 17:58:20.239406 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.239356 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-data\") pod \"aed79116-9b99-4427-814f-e8af212f5126\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " Nov 02 17:58:20.239406 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.239381 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-istio-token\" (UniqueName: \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-istio-token\") pod \"aed79116-9b99-4427-814f-e8af212f5126\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " Nov 02 17:58:20.239579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.239423 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-certs\") pod \"aed79116-9b99-4427-814f-e8af212f5126\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " Nov 02 17:58:20.239579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.239471 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aed79116-9b99-4427-814f-e8af212f5126-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed79116-9b99-4427-814f-e8af212f5126-istio-podinfo\") pod \"aed79116-9b99-4427-814f-e8af212f5126\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " Nov 02 17:58:20.239579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.239514 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-socket\") pod \"aed79116-9b99-4427-814f-e8af212f5126\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " Nov 02 17:58:20.239579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.239505 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-credential-socket" pod "aed79116-9b99-4427-814f-e8af212f5126" (UID: "aed79116-9b99-4427-814f-e8af212f5126"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.239579 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.239541 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed79116-9b99-4427-814f-e8af212f5126-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed79116-9b99-4427-814f-e8af212f5126-istiod-ca-cert\") pod \"aed79116-9b99-4427-814f-e8af212f5126\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " Nov 02 17:58:20.239839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.239600 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-kube-api-access-kp2v5\" (UniqueName: \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-kube-api-access-kp2v5\") pod \"aed79116-9b99-4427-814f-e8af212f5126\" (UID: \"aed79116-9b99-4427-814f-e8af212f5126\") " Nov 02 17:58:20.239839 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.239674 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-data" pod "aed79116-9b99-4427-814f-e8af212f5126" (UID: "aed79116-9b99-4427-814f-e8af212f5126"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.240091 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240056 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.240232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240098 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.240232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240114 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.240232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240126 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.240232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240139 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.240232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240136 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-socket" pod "aed79116-9b99-4427-814f-e8af212f5126" (UID: "aed79116-9b99-4427-814f-e8af212f5126"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.240232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240155 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.240232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240153 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-certs" pod "aed79116-9b99-4427-814f-e8af212f5126" (UID: "aed79116-9b99-4427-814f-e8af212f5126"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.240232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240168 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6069e34a-83a9-4812-85d4-ef0fd0e5173d-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.240232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240182 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6069e34a-83a9-4812-85d4-ef0fd0e5173d-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.240232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240195 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xk2n\" (UniqueName: \"kubernetes.io/projected/6069e34a-83a9-4812-85d4-ef0fd0e5173d-kube-api-access-4xk2n\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.240661 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.240315 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed79116-9b99-4427-814f-e8af212f5126-istiod-ca-cert" pod "aed79116-9b99-4427-814f-e8af212f5126" (UID: "aed79116-9b99-4427-814f-e8af212f5126"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:20.241824 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.241762 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-istio-token" pod "aed79116-9b99-4427-814f-e8af212f5126" (UID: "aed79116-9b99-4427-814f-e8af212f5126"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.242229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.242203 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-envoy" pod "aed79116-9b99-4427-814f-e8af212f5126" (UID: "aed79116-9b99-4427-814f-e8af212f5126"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:20.242229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.242209 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aed79116-9b99-4427-814f-e8af212f5126-istio-podinfo" pod "aed79116-9b99-4427-814f-e8af212f5126" (UID: "aed79116-9b99-4427-814f-e8af212f5126"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:20.242392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.242255 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-kube-api-access-kp2v5" pod "aed79116-9b99-4427-814f-e8af212f5126" (UID: "aed79116-9b99-4427-814f-e8af212f5126"). InnerVolumeSpecName "kube-api-access-kp2v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:20.273088 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.273046 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:20.347102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.341406 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.347102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.341450 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.347102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.341466 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed79116-9b99-4427-814f-e8af212f5126-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.347102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.341482 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.347102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.341495 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed79116-9b99-4427-814f-e8af212f5126-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.347102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.341510 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2v5\" (UniqueName: \"kubernetes.io/projected/aed79116-9b99-4427-814f-e8af212f5126-kube-api-access-kp2v5\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.347102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.341524 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed79116-9b99-4427-814f-e8af212f5126-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:58:20.391278 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.391244 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:20.526711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.526609 4310 generic.go:356] "Generic (PLEG): container finished" podID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerID="40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0" exitCode=0 Nov 02 17:58:20.526711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.526641 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" Nov 02 17:58:20.526711 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.526684 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" event={"ID":"6069e34a-83a9-4812-85d4-ef0fd0e5173d","Type":"ContainerDied","Data":"40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0"} Nov 02 17:58:20.527313 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.526742 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74" event={"ID":"6069e34a-83a9-4812-85d4-ef0fd0e5173d","Type":"ContainerDied","Data":"5783789197b4eac816f8e1ef90e8d2d94cbc8b1706c57ff5fbc86840c84dc857"} Nov 02 17:58:20.527313 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.526773 4310 scope.go:122] "RemoveContainer" containerID="40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0" Nov 02 17:58:20.528438 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.528417 4310 generic.go:356] "Generic (PLEG): container finished" podID="aed79116-9b99-4427-814f-e8af212f5126" containerID="8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4" exitCode=0 Nov 02 17:58:20.528552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.528463 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" Nov 02 17:58:20.528552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.528482 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" event={"ID":"aed79116-9b99-4427-814f-e8af212f5126","Type":"ContainerDied","Data":"8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4"} Nov 02 17:58:20.528552 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.528531 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc" event={"ID":"aed79116-9b99-4427-814f-e8af212f5126","Type":"ContainerDied","Data":"7321115f96a08a8a8dc156652360e311126efdf86020f5c2c01223f3e27c5484"} Nov 02 17:58:20.530179 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.530154 4310 generic.go:356] "Generic (PLEG): container finished" podID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerID="2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53" exitCode=0 Nov 02 17:58:20.530299 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.530194 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" Nov 02 17:58:20.530299 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.530219 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" event={"ID":"38d15b4f-697d-40ac-9299-fbf4b413127c","Type":"ContainerDied","Data":"2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53"} Nov 02 17:58:20.530299 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.530270 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w" event={"ID":"38d15b4f-697d-40ac-9299-fbf4b413127c","Type":"ContainerDied","Data":"0a5e1d6932d7c4e61fde4b2bd1a2291e6ff6fd72b221eece6a8ce60cb1811854"} Nov 02 17:58:20.532320 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.532295 4310 generic.go:356] "Generic (PLEG): container finished" podID="0253d943-e554-4580-b042-cabae8436768" containerID="5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6" exitCode=0 Nov 02 17:58:20.532476 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.532390 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" event={"ID":"0253d943-e554-4580-b042-cabae8436768","Type":"ContainerDied","Data":"5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6"} Nov 02 17:58:20.532476 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.532428 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" event={"ID":"0253d943-e554-4580-b042-cabae8436768","Type":"ContainerDied","Data":"579c053d784bcc61bed9c4dc0c46a350659676babef4a19bb190dc05783ffeb5"} Nov 02 17:58:20.532633 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.532494 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2" Nov 02 17:58:20.534652 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.534620 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:20.542354 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.542338 4310 scope.go:122] "RemoveContainer" containerID="40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0" Nov 02 17:58:20.542632 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:58:20.542616 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0\": container with ID starting with 40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0 not found: ID does not exist" containerID="40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0" Nov 02 17:58:20.542676 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.542641 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0"} err="failed to get container status \"40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0\": rpc error: code = NotFound desc = could not find container \"40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0\": container with ID starting with 40a4c5974ef419edbb151f6af22312016d62cd16b36c2064141793f037ee38d0 not found: ID does not exist" Nov 02 17:58:20.542676 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.542656 4310 scope.go:122] "RemoveContainer" containerID="8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4" Nov 02 17:58:20.554681 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.554657 4310 scope.go:122] "RemoveContainer" containerID="8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4" Nov 02 17:58:20.554987 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:58:20.554968 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4\": container with ID starting with 8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4 not found: ID does not exist" containerID="8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4" Nov 02 17:58:20.555053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.555004 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4"} err="failed to get container status \"8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4\": rpc error: code = NotFound desc = could not find container \"8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4\": container with ID starting with 8d9d12ab46b35f56d4a3e71c23a178ad261781dd3c37ea46a523046adcff56c4 not found: ID does not exist" Nov 02 17:58:20.555053 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.555024 4310 scope.go:122] "RemoveContainer" containerID="2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53" Nov 02 17:58:20.569236 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.569211 4310 scope.go:122] "RemoveContainer" containerID="2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53" Nov 02 17:58:20.569553 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:58:20.569532 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53\": container with ID starting with 2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53 not found: ID does not exist" containerID="2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53" Nov 02 17:58:20.569650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.569565 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53"} err="failed to get container status \"2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53\": rpc error: code = NotFound desc = could not find container \"2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53\": container with ID starting with 2ae3fda3fde972a89e786f8b0ffbbb7e423fe235c88db68b320fe424f348bb53 not found: ID does not exist" Nov 02 17:58:20.569650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.569582 4310 scope.go:122] "RemoveContainer" containerID="5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6" Nov 02 17:58:20.580497 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.580475 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w"] Nov 02 17:58:20.582218 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.581910 4310 scope.go:122] "RemoveContainer" containerID="5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6" Nov 02 17:58:20.582327 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:58:20.582285 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6\": container with ID starting with 5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6 not found: ID does not exist" containerID="5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6" Nov 02 17:58:20.582405 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.582318 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6"} err="failed to get container status \"5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6\": rpc error: code = NotFound desc = could not find container \"5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6\": container with ID starting with 5e8bdafeb704b2975f17211613a42065831c77d287899612973ef630296195f6 not found: ID does not exist" Nov 02 17:58:20.595061 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.595032 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-g824w"] Nov 02 17:58:20.615540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.615515 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74"] Nov 02 17:58:20.625856 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.625821 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-hss74"] Nov 02 17:58:20.641226 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.641198 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2"] Nov 02 17:58:20.651631 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.651606 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-p5vr2"] Nov 02 17:58:20.697014 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.695809 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:20.787473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.787389 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0253d943-e554-4580-b042-cabae8436768" path="/var/lib/kubelet/pods/0253d943-e554-4580-b042-cabae8436768/volumes" Nov 02 17:58:20.787941 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.787917 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" path="/var/lib/kubelet/pods/38d15b4f-697d-40ac-9299-fbf4b413127c/volumes" Nov 02 17:58:20.788503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.788483 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" path="/var/lib/kubelet/pods/6069e34a-83a9-4812-85d4-ef0fd0e5173d/volumes" Nov 02 17:58:20.789756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.789716 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc"] Nov 02 17:58:20.792909 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.792880 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2a295c-openshift-default-68d655ddb7-w2stc"] Nov 02 17:58:20.931912 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:20.931876 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:21.394170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:21.394135 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:22.182232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:22.182195 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:22.645417 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:22.645382 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:22.675051 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:22.675019 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed79116-9b99-4427-814f-e8af212f5126" path="/var/lib/kubelet/pods/aed79116-9b99-4427-814f-e8af212f5126/volumes" Nov 02 17:58:22.870834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:22.870793 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:23.031546 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:23.031515 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf"] Nov 02 17:58:23.036812 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:23.036786 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17bc4191-zvwkf"] Nov 02 17:58:23.752102 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:23.745687 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:24.483246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:24.483215 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:24.675112 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:24.675060 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8519447-24ff-4cb9-8856-47bb7fa04b60" path="/var/lib/kubelet/pods/c8519447-24ff-4cb9-8856-47bb7fa04b60/volumes" Nov 02 17:58:25.182336 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:25.182302 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:25.885403 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:25.885352 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:26.026062 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:26.026022 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv"] Nov 02 17:58:26.030114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:26.030088 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a15ff56-jjvdv"] Nov 02 17:58:26.670497 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:26.670459 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:26.674705 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:26.674675 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a19313-0559-4a24-956b-2b52cb2769bb" path="/var/lib/kubelet/pods/15a19313-0559-4a24-956b-2b52cb2769bb/volumes" Nov 02 17:58:27.029372 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:27.029340 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p"] Nov 02 17:58:27.033242 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:27.033220 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8abf3330-c4x5p"] Nov 02 17:58:27.374512 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:27.374416 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:28.081964 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:28.081933 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:28.674797 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:28.674757 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a919f38e-54f3-428a-8284-9aadff12c1c7" path="/var/lib/kubelet/pods/a919f38e-54f3-428a-8284-9aadff12c1c7/volumes" Nov 02 17:58:28.770557 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:28.770524 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:29.882393 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:29.882356 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:30.025956 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:30.025925 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl"] Nov 02 17:58:30.029708 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:30.029686 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b316709-5qldl"] Nov 02 17:58:30.166716 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:30.166638 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:30.675283 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:30.675248 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf2345b-ecfe-4ac8-8d4a-e9d07c495e35" path="/var/lib/kubelet/pods/faf2345b-ecfe-4ac8-8d4a-e9d07c495e35/volumes" Nov 02 17:58:30.929957 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:30.929877 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:31.672437 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:31.672404 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:32.443608 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:32.443543 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:33.082257 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:33.082203 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:33.524626 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:58:33.524583 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:58:33.786807 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:33.786710 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:34.473466 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:34.473438 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:35.261578 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:35.261536 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:35.983522 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:35.983475 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:36.683327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:36.683298 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:37.477727 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:37.477693 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:38.201970 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:38.201934 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:38.878692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:38.878655 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:39.575013 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:39.574967 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:40.362031 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:40.361991 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:40.985229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:40.985195 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:41.771504 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:41.771470 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:42.460262 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:42.460227 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:43.279043 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:43.279011 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:43.550510 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:43.550429 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:44.067574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:44.067533 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:44.255115 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:44.254418 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:44.976119 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:44.976054 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:45.789670 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:45.789630 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:46.482754 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:46.482713 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:47.037435 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:47.037399 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v"] Nov 02 17:58:47.045232 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:47.045204 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1217674-6sx2v"] Nov 02 17:58:47.363157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:47.362708 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:47.836609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:47.836578 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:47.963267 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:47.963232 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:48.029195 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:48.029146 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5"] Nov 02 17:58:48.036990 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:48.036958 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c154cd4e-mj7j5"] Nov 02 17:58:48.209405 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:48.209323 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:48.674503 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:48.674472 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d71e3bb-0fb8-4a62-9e43-36bff691191f" path="/var/lib/kubelet/pods/4d71e3bb-0fb8-4a62-9e43-36bff691191f/volumes" Nov 02 17:58:48.674956 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:48.674942 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafe078b-f2a5-4333-aedc-10ba3c204fa5" path="/var/lib/kubelet/pods/eafe078b-f2a5-4333-aedc-10ba3c204fa5/volumes" Nov 02 17:58:48.992409 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:48.992354 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:49.789825 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:49.789788 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:49.901142 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:49.901105 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:50.004351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.004313 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:50.572894 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.572860 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:50.630363 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.630326 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z"] Nov 02 17:58:50.630908 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.630889 4310 cpu_manager.go:414] "containerMap: removing container" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" Nov 02 17:58:50.631017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.630909 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" Nov 02 17:58:50.631017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.630923 4310 cpu_manager.go:414] "containerMap: removing container" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerName="istio-proxy" Nov 02 17:58:50.631017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.630932 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerName="istio-proxy" Nov 02 17:58:50.631017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.630949 4310 cpu_manager.go:414] "containerMap: removing container" podUID="af852c12-fc47-48e9-ac51-7418a37792d5" containerName="payload-processing" Nov 02 17:58:50.631017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.630957 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af852c12-fc47-48e9-ac51-7418a37792d5" containerName="payload-processing" Nov 02 17:58:50.631017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.630979 4310 cpu_manager.go:414] "containerMap: removing container" podUID="378c5f34-0627-4df2-8096-5bb1e582c650" containerName="payload-pre-processing" Nov 02 17:58:50.631017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.630989 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378c5f34-0627-4df2-8096-5bb1e582c650" containerName="payload-pre-processing" Nov 02 17:58:50.631017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631011 4310 cpu_manager.go:414] "containerMap: removing container" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" Nov 02 17:58:50.631017 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631019 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631034 4310 cpu_manager.go:414] "containerMap: removing container" podUID="605b73ac-e668-4fc1-ad46-a901f791d92b" containerName="payload-processing" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631042 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605b73ac-e668-4fc1-ad46-a901f791d92b" containerName="payload-processing" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631063 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d9f79edd-d030-40b3-a593-c6ea421cef43" containerName="revoke-keys" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631070 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f79edd-d030-40b3-a593-c6ea421cef43" containerName="revoke-keys" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631106 4310 cpu_manager.go:414] "containerMap: removing container" podUID="82ef9be3-7c5b-4407-b20b-277501637575" containerName="revoke-keys" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631114 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ef9be3-7c5b-4407-b20b-277501637575" containerName="revoke-keys" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631124 4310 cpu_manager.go:414] "containerMap: removing container" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631131 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631142 4310 cpu_manager.go:414] "containerMap: removing container" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631149 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631160 4310 cpu_manager.go:414] "containerMap: removing container" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631167 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631184 4310 cpu_manager.go:414] "containerMap: removing container" podUID="aed79116-9b99-4427-814f-e8af212f5126" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631191 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed79116-9b99-4427-814f-e8af212f5126" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631201 4310 cpu_manager.go:414] "containerMap: removing container" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631208 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631346 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="6069e34a-83a9-4812-85d4-ef0fd0e5173d" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631361 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="af852c12-fc47-48e9-ac51-7418a37792d5" containerName="payload-processing" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631374 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="378c5f34-0627-4df2-8096-5bb1e582c650" containerName="payload-pre-processing" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631385 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="0253d943-e554-4580-b042-cabae8436768" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631394 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="605b73ac-e668-4fc1-ad46-a901f791d92b" containerName="payload-processing" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631404 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f79edd-d030-40b3-a593-c6ea421cef43" containerName="revoke-keys" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631414 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631422 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ef9be3-7c5b-4407-b20b-277501637575" containerName="revoke-keys" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631432 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d15b4f-697d-40ac-9299-fbf4b413127c" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631441 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="557797b3-5dcf-4373-8a05-a8aa9fba451e" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631450 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7f8807-ebff-4a43-bb70-a161a1df954b" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631460 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed79116-9b99-4427-814f-e8af212f5126" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631469 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="153da82d-3725-4598-b171-e8b6b4205bf3" containerName="istio-proxy" Nov 02 17:58:50.631568 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.631480 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" Nov 02 17:58:50.637364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.637344 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 17:58:50.645543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.645514 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z"] Nov 02 17:58:50.647501 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.646518 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-263a5ecf\"" Nov 02 17:58:50.647718 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.647700 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-263a5ecf-dockercfg-prz55\"" Nov 02 17:58:50.689479 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.689439 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:50.803864 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.803821 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55hbk\" (UniqueName: \"kubernetes.io/projected/55222bef-563e-447e-af4b-ea8cc8fa190f-kube-api-access-55hbk\") pod \"payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z\" (UID: \"55222bef-563e-447e-af4b-ea8cc8fa190f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 17:58:50.804293 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.804268 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55222bef-563e-447e-af4b-ea8cc8fa190f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z\" (UID: \"55222bef-563e-447e-af4b-ea8cc8fa190f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 17:58:50.906267 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.906174 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hbk\" (UniqueName: \"kubernetes.io/projected/55222bef-563e-447e-af4b-ea8cc8fa190f-kube-api-access-55hbk\") pod \"payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z\" (UID: \"55222bef-563e-447e-af4b-ea8cc8fa190f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 17:58:50.906267 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.906247 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55222bef-563e-447e-af4b-ea8cc8fa190f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z\" (UID: \"55222bef-563e-447e-af4b-ea8cc8fa190f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 17:58:50.906977 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.906959 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55222bef-563e-447e-af4b-ea8cc8fa190f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z\" (UID: \"55222bef-563e-447e-af4b-ea8cc8fa190f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 17:58:50.917833 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.917806 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hbk\" (UniqueName: \"kubernetes.io/projected/55222bef-563e-447e-af4b-ea8cc8fa190f-kube-api-access-55hbk\") pod \"payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z\" (UID: \"55222bef-563e-447e-af4b-ea8cc8fa190f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 17:58:50.957574 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:50.957543 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 17:58:51.270197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:51.270167 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z"] Nov 02 17:58:51.270571 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:58:51.270547 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55222bef_563e_447e_af4b_ea8cc8fa190f.slice/crio-74f7479f9662a009f71838c2f4f23bac2563a58776660f5397e61c78cc74a876 WatchSource:0}: Error finding container 74f7479f9662a009f71838c2f4f23bac2563a58776660f5397e61c78cc74a876: Status 404 returned error can't find the container with id 74f7479f9662a009f71838c2f4f23bac2563a58776660f5397e61c78cc74a876 Nov 02 17:58:51.688070 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:51.687994 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:51.835046 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:51.835008 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" event={"ID":"55222bef-563e-447e-af4b-ea8cc8fa190f","Type":"ContainerStarted","Data":"8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17"} Nov 02 17:58:51.835257 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:51.835103 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 17:58:51.835257 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:51.835124 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" event={"ID":"55222bef-563e-447e-af4b-ea8cc8fa190f","Type":"ContainerStarted","Data":"74f7479f9662a009f71838c2f4f23bac2563a58776660f5397e61c78cc74a876"} Nov 02 17:58:51.857763 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:51.857717 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" podStartSLOduration=1.8577020800000001 podStartE2EDuration="1.85770208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:51.852304371 +0000 UTC m=+1129.295387007" watchObservedRunningTime="2026-11-02 17:58:51.85770208 +0000 UTC m=+1129.300784760" Nov 02 17:58:52.032667 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:52.032634 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj"] Nov 02 17:58:52.036640 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:52.036612 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33d927b-jswxj"] Nov 02 17:58:52.547098 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:52.546617 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:52.676412 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:52.676378 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f67fab-d093-4478-a09b-8d6e2921d0c0" path="/var/lib/kubelet/pods/25f67fab-d093-4478-a09b-8d6e2921d0c0/volumes" Nov 02 17:58:53.181889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:53.181851 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:53.876814 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:53.876779 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:54.580459 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:54.580420 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:55.277184 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:55.277149 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:56.033171 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:56.033134 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6"] Nov 02 17:58:56.038183 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:56.038156 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9942a114-gfsk6"] Nov 02 17:58:56.042204 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:56.042166 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52"] Nov 02 17:58:56.045641 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:56.045620 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16e4609-wzp52"] Nov 02 17:58:56.067278 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:56.067245 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:56.674291 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:56.674257 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e642a3b-628a-4120-8db7-77005882e0af" path="/var/lib/kubelet/pods/6e642a3b-628a-4120-8db7-77005882e0af/volumes" Nov 02 17:58:56.674898 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:56.674880 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba6d31f-770c-4907-84e9-ce759ff12719" path="/var/lib/kubelet/pods/8ba6d31f-770c-4907-84e9-ce759ff12719/volumes" Nov 02 17:58:56.879857 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:56.879815 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:57.595229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:57.595200 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:58.268250 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:58.268209 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:58.970819 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:58.970783 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:58:59.773656 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:58:59.773624 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:00.027744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:00.027664 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7"] Nov 02 17:59:00.031331 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:00.031308 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7427c3d9-xk7p7"] Nov 02 17:59:00.675283 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:00.675252 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1acfc3a9-75e8-461d-a29b-69eb0a4167f6" path="/var/lib/kubelet/pods/1acfc3a9-75e8-461d-a29b-69eb0a4167f6/volumes" Nov 02 17:59:00.679865 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:00.679833 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:01.543376 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:01.543343 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:02.171323 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:02.171289 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:02.778971 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:02.778934 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:02.850027 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:02.848961 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 17:59:03.013195 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:03.013139 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:03.524166 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:59:03.524131 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:59:03.710754 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:03.710720 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:03.871593 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:03.871517 4310 scope.go:122] "RemoveContainer" containerID="be8536cd41f6c4e492cba5011ef87c72fea8ace12466b1b05c2d84b9c7d3a240" Nov 02 17:59:03.882453 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:03.882432 4310 scope.go:122] "RemoveContainer" containerID="45592a15844cbe96f1b6cfefa2c6e1a9fe60fc4aca77f20d8c52ac720233c25c" Nov 02 17:59:03.890477 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:03.890451 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:03.893127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:03.893111 4310 scope.go:122] "RemoveContainer" containerID="58ae626fb4e7df3a72e5ec9477870dde5283f99ff6e5c4747166f869930827c9" Nov 02 17:59:03.908370 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:03.908342 4310 scope.go:122] "RemoveContainer" containerID="9b09e444a68b50f6a49a876f28a33e248c61bc508d7b487d522bb9085d5c5464" Nov 02 17:59:04.254149 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:04.254117 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:04.382640 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:04.382609 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:05.026048 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:05.026010 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w"] Nov 02 17:59:05.029780 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:05.029754 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00aa2fff-dpp6w"] Nov 02 17:59:05.064857 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:05.064832 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:05.777450 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:05.777417 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:06.567297 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:06.567256 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:06.674338 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:06.674304 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a2dccb-8da1-4236-a84d-51f0319d1a8e" path="/var/lib/kubelet/pods/00a2dccb-8da1-4236-a84d-51f0319d1a8e/volumes" Nov 02 17:59:07.279570 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:07.279534 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:07.924663 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:07.924623 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:08.694382 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:08.694352 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:09.497516 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:09.497484 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:10.252576 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:10.252483 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:10.902892 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:10.902853 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:11.042110 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:11.035233 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:11.685279 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:11.685246 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:12.500603 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:12.500567 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:13.281557 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:13.281520 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:13.871983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:13.871880 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:14.685573 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:14.685523 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:15.380500 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:15.380465 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:16.090302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:16.090269 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:16.787241 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:16.787210 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:17.032065 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:17.032032 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn"] Nov 02 17:59:17.035740 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:17.035711 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbaaebc6-5smcn"] Nov 02 17:59:17.581989 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:17.581953 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:18.294447 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:18.294407 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:18.674392 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:18.674308 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208e4394-235b-431d-9e76-a06f2feb9f28" path="/var/lib/kubelet/pods/208e4394-235b-431d-9e76-a06f2feb9f28/volumes" Nov 02 17:59:19.026160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:19.026111 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5"] Nov 02 17:59:19.029662 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:19.029635 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725ef6d4-4h4z5"] Nov 02 17:59:19.064994 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:19.064967 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:19.777753 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:19.777720 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:20.026731 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:20.026698 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd"] Nov 02 17:59:20.031029 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:20.030958 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5ea585-bbkkd"] Nov 02 17:59:20.472765 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:20.472718 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:20.675923 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:20.675892 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e577ac-e4cf-4b98-8243-332c9be8c7f3" path="/var/lib/kubelet/pods/a4e577ac-e4cf-4b98-8243-332c9be8c7f3/volumes" Nov 02 17:59:20.676366 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:20.676351 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb3cc13-1c8c-4183-ab6a-d2310b054d45" path="/var/lib/kubelet/pods/abb3cc13-1c8c-4183-ab6a-d2310b054d45/volumes" Nov 02 17:59:21.025923 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:21.025888 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br"] Nov 02 17:59:21.029318 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:21.029294 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63487414-4d7br"] Nov 02 17:59:21.171091 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:21.171045 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:21.880702 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:21.880670 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:22.667786 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:22.667751 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:22.674674 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:22.674647 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e49a42-496d-4efe-87ac-70a9bfa43dfc" path="/var/lib/kubelet/pods/d4e49a42-496d-4efe-87ac-70a9bfa43dfc/volumes" Nov 02 17:59:23.369473 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:23.369432 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:24.166319 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:24.166286 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:24.890804 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:24.890766 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:25.567464 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:25.567430 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:26.267781 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:26.267740 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:26.960623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:26.960586 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:27.054157 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:27.054124 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r"] Nov 02 17:59:27.058186 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:27.057917 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0782c3a-sgn8r"] Nov 02 17:59:27.666454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:27.666424 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:28.025152 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:28.025120 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw"] Nov 02 17:59:28.028502 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:28.028465 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a6f6ed-5mqzw"] Nov 02 17:59:28.385289 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:28.385186 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:28.673889 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:28.673809 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b62f9b-f0af-4474-a52e-5fb4a3c39e4d" path="/var/lib/kubelet/pods/35b62f9b-f0af-4474-a52e-5fb4a3c39e4d/volumes" Nov 02 17:59:28.674307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:28.674293 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770cd47b-7a5d-4885-bae2-98948b2ddedd" path="/var/lib/kubelet/pods/770cd47b-7a5d-4885-bae2-98948b2ddedd/volumes" Nov 02 17:59:29.030631 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:29.030597 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z"] Nov 02 17:59:29.033902 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:29.033880 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd13214-dn79z"] Nov 02 17:59:29.076347 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:29.076314 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:29.789616 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:29.789581 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:30.028307 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:30.028264 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7"] Nov 02 17:59:30.031682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:30.031653 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d8a44d-mn2h7"] Nov 02 17:59:30.496921 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:30.496883 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:30.676491 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:30.676455 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c519182-e8de-4e4a-8668-af65f7dea9b6" path="/var/lib/kubelet/pods/4c519182-e8de-4e4a-8668-af65f7dea9b6/volumes" Nov 02 17:59:30.676923 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:30.676908 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3017323-41c0-4cde-9edb-ba9a6185351f" path="/var/lib/kubelet/pods/c3017323-41c0-4cde-9edb-ba9a6185351f/volumes" Nov 02 17:59:31.260216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:31.260171 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:31.940123 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:31.940044 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:32.777327 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:32.777297 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:33.023517 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:33.023487 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn"] Nov 02 17:59:33.026717 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:33.026690 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e035996-zkkdn"] Nov 02 17:59:33.393467 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:33.393433 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:33.523385 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:59:33.523347 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 17:59:34.093827 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:34.093768 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:34.674527 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:34.674494 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c" path="/var/lib/kubelet/pods/fb3145a2-86eb-49fb-8cd7-2a5f9ef7522c/volumes" Nov 02 17:59:34.896933 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:34.896896 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:35.024990 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:35.024952 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv"] Nov 02 17:59:35.028170 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:35.028141 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ee2a68-wwtkv"] Nov 02 17:59:35.690834 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:35.690804 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:36.380702 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:36.380666 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:36.676458 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:36.676384 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e4a96e6-8cd1-4531-bf13-49de4b1e6c69" path="/var/lib/kubelet/pods/6e4a96e6-8cd1-4531-bf13-49de4b1e6c69/volumes" Nov 02 17:59:37.062798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.062757 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:37.215798 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.215767 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:37.373197 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.373116 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv"] Nov 02 17:59:37.373680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.373658 4310 cpu_manager.go:414] "containerMap: removing container" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" Nov 02 17:59:37.373680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.373679 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4106a2f4-2564-4e87-a040-0c62ecaf2fb8" containerName="istio-proxy" Nov 02 17:59:37.378329 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.378310 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:37.383160 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.383136 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:59:37.383921 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.383900 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:59:37.389127 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.389102 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv"] Nov 02 17:59:37.428672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.428636 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:37.428845 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.428752 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:37.428923 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.428836 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgskh\" (UniqueName: \"kubernetes.io/projected/2878b8cb-1876-43e5-902d-a4dfd302ef8f-kube-api-access-vgskh\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:37.530273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.530241 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:37.530273 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.530284 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgskh\" (UniqueName: \"kubernetes.io/projected/2878b8cb-1876-43e5-902d-a4dfd302ef8f-kube-api-access-vgskh\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:37.530490 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.530419 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:37.531059 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.531040 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:37.532644 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.532619 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:37.538692 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.538671 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgskh\" (UniqueName: \"kubernetes.io/projected/2878b8cb-1876-43e5-902d-a4dfd302ef8f-kube-api-access-vgskh\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:37.694369 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.694288 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:37.959363 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.959277 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:37.975454 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:37.975421 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv"] Nov 02 17:59:37.976526 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:59:37.976483 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2878b8cb_1876_43e5_902d_a4dfd302ef8f.slice/crio-957384b8a8fca36a651a715d458f5584fa4de6eefff471f4080813d308bde7ad WatchSource:0}: Error finding container 957384b8a8fca36a651a715d458f5584fa4de6eefff471f4080813d308bde7ad: Status 404 returned error can't find the container with id 957384b8a8fca36a651a715d458f5584fa4de6eefff471f4080813d308bde7ad Nov 02 17:59:38.325260 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:38.325224 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" event={"ID":"2878b8cb-1876-43e5-902d-a4dfd302ef8f","Type":"ContainerStarted","Data":"957384b8a8fca36a651a715d458f5584fa4de6eefff471f4080813d308bde7ad"} Nov 02 17:59:38.672495 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:38.672406 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:39.334603 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:39.334562 4310 generic.go:356] "Generic (PLEG): container finished" podID="2878b8cb-1876-43e5-902d-a4dfd302ef8f" containerID="7c06fcc4758b3d3cd2463f21af3d6ec64bf724ac32121aaf68236c74fca859d9" exitCode=0 Nov 02 17:59:39.334968 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:39.334649 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" event={"ID":"2878b8cb-1876-43e5-902d-a4dfd302ef8f","Type":"ContainerDied","Data":"7c06fcc4758b3d3cd2463f21af3d6ec64bf724ac32121aaf68236c74fca859d9"} Nov 02 17:59:39.532101 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:39.531351 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:40.180494 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.180457 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:40.484782 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.484760 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:40.571966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.571932 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-xks-ca\") pod \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " Nov 02 17:59:40.571966 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.571970 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2878b8cb-1876-43e5-902d-a4dfd302ef8f-kube-api-access-vgskh\" (UniqueName: \"kubernetes.io/projected/2878b8cb-1876-43e5-902d-a4dfd302ef8f-kube-api-access-vgskh\") pod \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " Nov 02 17:59:40.572214 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.572061 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-openshift-ca\") pod \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\" (UID: \"2878b8cb-1876-43e5-902d-a4dfd302ef8f\") " Nov 02 17:59:40.572482 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.572452 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-openshift-ca" pod "2878b8cb-1876-43e5-902d-a4dfd302ef8f" (UID: "2878b8cb-1876-43e5-902d-a4dfd302ef8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:59:40.574114 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.574069 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-xks-ca" pod "2878b8cb-1876-43e5-902d-a4dfd302ef8f" (UID: "2878b8cb-1876-43e5-902d-a4dfd302ef8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:59:40.574216 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.574142 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2878b8cb-1876-43e5-902d-a4dfd302ef8f-kube-api-access-vgskh" pod "2878b8cb-1876-43e5-902d-a4dfd302ef8f" (UID: "2878b8cb-1876-43e5-902d-a4dfd302ef8f"). InnerVolumeSpecName "kube-api-access-vgskh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:40.673044 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.673012 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:40.673246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.673043 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2878b8cb-1876-43e5-902d-a4dfd302ef8f-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:40.673246 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.673073 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgskh\" (UniqueName: \"kubernetes.io/projected/2878b8cb-1876-43e5-902d-a4dfd302ef8f-kube-api-access-vgskh\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:40.878478 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:40.878446 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:41.351469 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:41.351442 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" Nov 02 17:59:41.351469 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:41.351449 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv" event={"ID":"2878b8cb-1876-43e5-902d-a4dfd302ef8f","Type":"ContainerDied","Data":"957384b8a8fca36a651a715d458f5584fa4de6eefff471f4080813d308bde7ad"} Nov 02 17:59:41.353637 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:41.351494 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="957384b8a8fca36a651a715d458f5584fa4de6eefff471f4080813d308bde7ad" Nov 02 17:59:41.632002 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:41.631916 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:42.178418 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:42.178380 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz"] Nov 02 17:59:42.184540 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:42.184506 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz"] Nov 02 17:59:42.291778 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:42.291731 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:42.399790 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:42.399718 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:43.068764 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:43.068735 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:43.168744 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:43.168715 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:43.875182 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:43.875141 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:44.479543 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:44.479503 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:45.286061 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:45.286024 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:45.983921 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:45.983882 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:46.051472 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:46.051419 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4"] Nov 02 17:59:46.058186 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:46.058154 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q"] Nov 02 17:59:46.063901 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:46.063864 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2032de45-cfpb4"] Nov 02 17:59:46.069593 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:46.069530 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2042f8ad-br72q"] Nov 02 17:59:46.673890 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:46.673857 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12435051-cef8-4a9b-b749-b1c6ba61e6c9" path="/var/lib/kubelet/pods/12435051-cef8-4a9b-b749-b1c6ba61e6c9/volumes" Nov 02 17:59:46.674346 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:46.674331 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8eedda-d223-48e0-8c4e-9f109a62fd12" path="/var/lib/kubelet/pods/9b8eedda-d223-48e0-8c4e-9f109a62fd12/volumes" Nov 02 17:59:46.687052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:46.687021 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:47.494709 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:47.494671 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l"] Nov 02 17:59:47.575986 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:47.575951 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:47.684420 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:47.684374 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:47.943662 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:47.943577 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:48.023988 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.023953 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc"] Nov 02 17:59:48.026884 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.026863 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c41026-8bjgc"] Nov 02 17:59:48.173063 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.173027 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4"] Nov 02 17:59:48.173530 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.173517 4310 cpu_manager.go:414] "containerMap: removing container" podUID="2878b8cb-1876-43e5-902d-a4dfd302ef8f" containerName="revoke-keys" Nov 02 17:59:48.173583 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.173532 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2878b8cb-1876-43e5-902d-a4dfd302ef8f" containerName="revoke-keys" Nov 02 17:59:48.173647 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.173638 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="2878b8cb-1876-43e5-902d-a4dfd302ef8f" containerName="revoke-keys" Nov 02 17:59:48.178303 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.178285 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:48.184202 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.184177 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:59:48.184570 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.184541 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 17:59:48.195219 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.195183 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx"] Nov 02 17:59:48.211210 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.211180 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe2dba4-jxgg4\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:48.211365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.211221 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe2dba4-jxgg4\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:48.211365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.211246 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:48.211365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.211279 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54zg6\" (UniqueName: \"kubernetes.io/projected/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-kube-api-access-54zg6\") pod \"maas-api-revoke-sub-7fe2dba4-jxgg4\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:48.213650 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.213631 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4"] Nov 02 17:59:48.218886 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.218861 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx"] Nov 02 17:59:48.312351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.312306 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f852g\" (UniqueName: \"kubernetes.io/projected/fa58c510-6474-462b-9455-d676d33e6784-kube-api-access-f852g\") pod \"maas-api-revoke-sub-b18460d9-9fqzx\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:48.312351 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.312356 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa58c510-6474-462b-9455-d676d33e6784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18460d9-9fqzx\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:48.312590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.312483 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa58c510-6474-462b-9455-d676d33e6784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18460d9-9fqzx\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:48.312590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.312516 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe2dba4-jxgg4\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:48.312590 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.312534 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe2dba4-jxgg4\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:48.312704 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.312619 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zg6\" (UniqueName: \"kubernetes.io/projected/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-kube-api-access-54zg6\") pod \"maas-api-revoke-sub-7fe2dba4-jxgg4\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:48.313243 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.313225 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe2dba4-jxgg4\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:48.315047 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.315030 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe2dba4-jxgg4\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:48.322901 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.322870 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zg6\" (UniqueName: \"kubernetes.io/projected/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-kube-api-access-54zg6\") pod \"maas-api-revoke-sub-7fe2dba4-jxgg4\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:48.413902 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.413848 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f852g\" (UniqueName: \"kubernetes.io/projected/fa58c510-6474-462b-9455-d676d33e6784-kube-api-access-f852g\") pod \"maas-api-revoke-sub-b18460d9-9fqzx\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:48.413902 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.413906 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa58c510-6474-462b-9455-d676d33e6784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18460d9-9fqzx\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:48.414179 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.413989 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa58c510-6474-462b-9455-d676d33e6784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18460d9-9fqzx\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:48.414672 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.414650 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa58c510-6474-462b-9455-d676d33e6784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18460d9-9fqzx\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:48.416443 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.416422 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa58c510-6474-462b-9455-d676d33e6784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18460d9-9fqzx\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:48.423162 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.423141 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f852g\" (UniqueName: \"kubernetes.io/projected/fa58c510-6474-462b-9455-d676d33e6784-kube-api-access-f852g\") pod \"maas-api-revoke-sub-b18460d9-9fqzx\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:48.462185 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.462154 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:48.512097 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.510596 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:48.526581 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.526549 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:48.674610 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.674568 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ff0895-1873-4e12-b960-b38d6a746da8" path="/var/lib/kubelet/pods/57ff0895-1873-4e12-b960-b38d6a746da8/volumes" Nov 02 17:59:48.834682 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.834653 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4"] Nov 02 17:59:48.835184 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:59:48.835156 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6b4ee4_b67f_4c91_9db2_103f7c00c052.slice/crio-abb06a32262e8485ef1882bc756eb090cb52bed174b63f8dac625968eaf22541 WatchSource:0}: Error finding container abb06a32262e8485ef1882bc756eb090cb52bed174b63f8dac625968eaf22541: Status 404 returned error can't find the container with id abb06a32262e8485ef1882bc756eb090cb52bed174b63f8dac625968eaf22541 Nov 02 17:59:48.839280 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:48.839229 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx"] Nov 02 17:59:48.839604 ip-10-0-135-9 kubenswrapper[4310]: W1102 17:59:48.839568 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa58c510_6474_462b_9455_d676d33e6784.slice/crio-993b8c87e3b1bf99ec6409970bf961e5027be10fbe0fde765dd63a0216edc0fa WatchSource:0}: Error finding container 993b8c87e3b1bf99ec6409970bf961e5027be10fbe0fde765dd63a0216edc0fa: Status 404 returned error can't find the container with id 993b8c87e3b1bf99ec6409970bf961e5027be10fbe0fde765dd63a0216edc0fa Nov 02 17:59:49.245339 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:49.244335 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:49.469499 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:49.469461 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" event={"ID":"7f6b4ee4-b67f-4c91-9db2-103f7c00c052","Type":"ContainerStarted","Data":"f9d47020381e4741480563b9ecf03208cd9b8c6c2c82b449dc566963afb26b46"} Nov 02 17:59:49.469670 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:49.469523 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" event={"ID":"7f6b4ee4-b67f-4c91-9db2-103f7c00c052","Type":"ContainerStarted","Data":"abb06a32262e8485ef1882bc756eb090cb52bed174b63f8dac625968eaf22541"} Nov 02 17:59:49.471299 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:49.471266 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" event={"ID":"fa58c510-6474-462b-9455-d676d33e6784","Type":"ContainerStarted","Data":"2b14b468375c7de31e6bf1c9b9179e16327ca24c141e5009e139afe1455579a9"} Nov 02 17:59:49.471461 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:49.471323 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" event={"ID":"fa58c510-6474-462b-9455-d676d33e6784","Type":"ContainerStarted","Data":"993b8c87e3b1bf99ec6409970bf961e5027be10fbe0fde765dd63a0216edc0fa"} Nov 02 17:59:49.490785 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:49.490735 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" podStartSLOduration=1.490721599 podStartE2EDuration="1.490721599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:49.488156395 +0000 UTC m=+1186.931239033" watchObservedRunningTime="2026-11-02 17:59:49.490721599 +0000 UTC m=+1186.933804296" Nov 02 17:59:49.508379 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:49.508329 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" podStartSLOduration=1.508317592 podStartE2EDuration="1.508317592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:49.505803924 +0000 UTC m=+1186.948886561" watchObservedRunningTime="2026-11-02 17:59:49.508317592 +0000 UTC m=+1186.951400227" Nov 02 17:59:49.902934 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:49.902894 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:50.480671 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:50.480636 4310 generic.go:356] "Generic (PLEG): container finished" podID="7f6b4ee4-b67f-4c91-9db2-103f7c00c052" containerID="f9d47020381e4741480563b9ecf03208cd9b8c6c2c82b449dc566963afb26b46" exitCode=0 Nov 02 17:59:50.480882 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:50.480720 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" event={"ID":"7f6b4ee4-b67f-4c91-9db2-103f7c00c052","Type":"ContainerDied","Data":"f9d47020381e4741480563b9ecf03208cd9b8c6c2c82b449dc566963afb26b46"} Nov 02 17:59:50.482260 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:50.482239 4310 generic.go:356] "Generic (PLEG): container finished" podID="fa58c510-6474-462b-9455-d676d33e6784" containerID="2b14b468375c7de31e6bf1c9b9179e16327ca24c141e5009e139afe1455579a9" exitCode=0 Nov 02 17:59:50.482378 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:50.482326 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" event={"ID":"fa58c510-6474-462b-9455-d676d33e6784","Type":"ContainerDied","Data":"2b14b468375c7de31e6bf1c9b9179e16327ca24c141e5009e139afe1455579a9"} Nov 02 17:59:50.615655 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:50.615622 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:51.568948 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.568916 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:51.706095 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.706060 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:51.708921 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.708900 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:51.799207 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.799122 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa58c510-6474-462b-9455-d676d33e6784-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa58c510-6474-462b-9455-d676d33e6784-maas-api-openshift-ca\") pod \"fa58c510-6474-462b-9455-d676d33e6784\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " Nov 02 17:59:51.799364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.799217 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa58c510-6474-462b-9455-d676d33e6784-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa58c510-6474-462b-9455-d676d33e6784-maas-api-xks-ca\") pod \"fa58c510-6474-462b-9455-d676d33e6784\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " Nov 02 17:59:51.799364 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.799285 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-openshift-ca\") pod \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " Nov 02 17:59:51.799474 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.799404 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-xks-ca\") pod \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " Nov 02 17:59:51.799557 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.799531 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa58c510-6474-462b-9455-d676d33e6784-maas-api-openshift-ca" pod "fa58c510-6474-462b-9455-d676d33e6784" (UID: "fa58c510-6474-462b-9455-d676d33e6784"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:59:51.799678 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.799599 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-kube-api-access-54zg6\" (UniqueName: \"kubernetes.io/projected/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-kube-api-access-54zg6\") pod \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\" (UID: \"7f6b4ee4-b67f-4c91-9db2-103f7c00c052\") " Nov 02 17:59:51.799678 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.799622 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-openshift-ca" pod "7f6b4ee4-b67f-4c91-9db2-103f7c00c052" (UID: "7f6b4ee4-b67f-4c91-9db2-103f7c00c052"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:59:51.799678 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.799636 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa58c510-6474-462b-9455-d676d33e6784-kube-api-access-f852g\" (UniqueName: \"kubernetes.io/projected/fa58c510-6474-462b-9455-d676d33e6784-kube-api-access-f852g\") pod \"fa58c510-6474-462b-9455-d676d33e6784\" (UID: \"fa58c510-6474-462b-9455-d676d33e6784\") " Nov 02 17:59:51.800203 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.800166 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:51.800203 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.800195 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa58c510-6474-462b-9455-d676d33e6784-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:51.801700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.801640 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-xks-ca" pod "7f6b4ee4-b67f-4c91-9db2-103f7c00c052" (UID: "7f6b4ee4-b67f-4c91-9db2-103f7c00c052"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:59:51.801700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.801661 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa58c510-6474-462b-9455-d676d33e6784-kube-api-access-f852g" pod "fa58c510-6474-462b-9455-d676d33e6784" (UID: "fa58c510-6474-462b-9455-d676d33e6784"). InnerVolumeSpecName "kube-api-access-f852g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:51.801700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.801662 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa58c510-6474-462b-9455-d676d33e6784-maas-api-xks-ca" pod "fa58c510-6474-462b-9455-d676d33e6784" (UID: "fa58c510-6474-462b-9455-d676d33e6784"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:59:51.801700 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.801693 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-kube-api-access-54zg6" pod "7f6b4ee4-b67f-4c91-9db2-103f7c00c052" (UID: "7f6b4ee4-b67f-4c91-9db2-103f7c00c052"). InnerVolumeSpecName "kube-api-access-54zg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:51.901052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.901015 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54zg6\" (UniqueName: \"kubernetes.io/projected/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-kube-api-access-54zg6\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:51.901052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.901045 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f852g\" (UniqueName: \"kubernetes.io/projected/fa58c510-6474-462b-9455-d676d33e6784-kube-api-access-f852g\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:51.901052 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.901056 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa58c510-6474-462b-9455-d676d33e6784-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:51.901302 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:51.901067 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6b4ee4-b67f-4c91-9db2-103f7c00c052-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:52.277926 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:52.277894 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:52.495815 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:52.495771 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" podUID="12043acc-60b0-48d1-9164-127fc5215f28" containerName="payload-processing" containerID="cri-o://ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57" gracePeriod=25 Nov 02 17:59:52.529345 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:52.529265 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" Nov 02 17:59:52.529507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:52.529264 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4" event={"ID":"7f6b4ee4-b67f-4c91-9db2-103f7c00c052","Type":"ContainerDied","Data":"abb06a32262e8485ef1882bc756eb090cb52bed174b63f8dac625968eaf22541"} Nov 02 17:59:52.529507 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:52.529422 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abb06a32262e8485ef1882bc756eb090cb52bed174b63f8dac625968eaf22541" Nov 02 17:59:52.530852 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:52.530831 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" Nov 02 17:59:52.530983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:52.530916 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx" event={"ID":"fa58c510-6474-462b-9455-d676d33e6784","Type":"ContainerDied","Data":"993b8c87e3b1bf99ec6409970bf961e5027be10fbe0fde765dd63a0216edc0fa"} Nov 02 17:59:52.530983 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:52.530954 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="993b8c87e3b1bf99ec6409970bf961e5027be10fbe0fde765dd63a0216edc0fa" Nov 02 17:59:53.079871 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:53.079834 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:53.846832 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:53.846741 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:54.465745 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:54.465712 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:55.204376 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:55.204343 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:55.865835 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:55.865795 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:56.025350 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:56.025308 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v"] Nov 02 17:59:56.028439 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:56.028417 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb309a27-bdm9v"] Nov 02 17:59:56.674710 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:56.674674 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481151a8-e4d4-4a0f-ae48-68102f129a23" path="/var/lib/kubelet/pods/481151a8-e4d4-4a0f-ae48-68102f129a23/volumes" Nov 02 17:59:56.678957 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:56.678930 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:57.200519 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.200475 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" containerID="cri-o://3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d" gracePeriod=45 Nov 02 17:59:57.205145 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.205105 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" containerID="cri-o://3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2" gracePeriod=45 Nov 02 17:59:57.362502 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.362467 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:57.504181 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.504147 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:59:57.555831 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.555803 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:59:57.584042 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.584009 4310 generic.go:356] "Generic (PLEG): container finished" podID="784056a6-b287-4aa3-af0e-b1175a88200b" containerID="3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d" exitCode=0 Nov 02 17:59:57.584274 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.584256 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" Nov 02 17:59:57.584609 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.584588 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" event={"ID":"784056a6-b287-4aa3-af0e-b1175a88200b","Type":"ContainerDied","Data":"3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d"} Nov 02 17:59:57.584680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.584631 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz" event={"ID":"784056a6-b287-4aa3-af0e-b1175a88200b","Type":"ContainerDied","Data":"d54998ccbf55368f4df6b038af3f3d4b08dd61807e1d86b128c427b539543770"} Nov 02 17:59:57.584680 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.584654 4310 scope.go:122] "RemoveContainer" containerID="3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d" Nov 02 17:59:57.589391 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.589308 4310 generic.go:356] "Generic (PLEG): container finished" podID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerID="3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2" exitCode=0 Nov 02 17:59:57.589391 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.589350 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" Nov 02 17:59:57.589539 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.589350 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" event={"ID":"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8","Type":"ContainerDied","Data":"3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2"} Nov 02 17:59:57.589539 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.589507 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz" event={"ID":"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8","Type":"ContainerDied","Data":"27afaf03e901ac4173d071af5fd9094d556071493d9c8c93b7de27f9eaa5528c"} Nov 02 17:59:57.602268 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.602248 4310 scope.go:122] "RemoveContainer" containerID="3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d" Nov 02 17:59:57.602629 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:59:57.602606 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d\": container with ID starting with 3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d not found: ID does not exist" containerID="3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d" Nov 02 17:59:57.602734 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.602637 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d"} err="failed to get container status \"3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d\": rpc error: code = NotFound desc = could not find container \"3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d\": container with ID starting with 3ee51e39e3ed9f17f683455cda1b768384b949253d44bc567787337436b1fd1d not found: ID does not exist" Nov 02 17:59:57.602734 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.602657 4310 scope.go:122] "RemoveContainer" containerID="3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2" Nov 02 17:59:57.613394 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.613366 4310 scope.go:122] "RemoveContainer" containerID="3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2" Nov 02 17:59:57.613678 ip-10-0-135-9 kubenswrapper[4310]: E1102 17:59:57.613651 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2\": container with ID starting with 3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2 not found: ID does not exist" containerID="3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2" Nov 02 17:59:57.613748 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.613679 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2"} err="failed to get container status \"3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2\": rpc error: code = NotFound desc = could not find container \"3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2\": container with ID starting with 3b23f750b9f4251d3e9f62baee9625165ea2393dda9a9f02dbdfafc91509bfd2 not found: ID does not exist" Nov 02 17:59:57.681721 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.681686 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-tmp-dir\") pod \"784056a6-b287-4aa3-af0e-b1175a88200b\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " Nov 02 17:59:57.681891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.681749 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784056a6-b287-4aa3-af0e-b1175a88200b-tls-certs\" (UniqueName: \"kubernetes.io/secret/784056a6-b287-4aa3-af0e-b1175a88200b-tls-certs\") pod \"784056a6-b287-4aa3-af0e-b1175a88200b\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " Nov 02 17:59:57.681891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.681788 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-home\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-home\") pod \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " Nov 02 17:59:57.681891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.681818 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-model-cache\") pod \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " Nov 02 17:59:57.681891 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.681871 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-kube-api-access-rt2vl\" (UniqueName: \"kubernetes.io/projected/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-kube-api-access-rt2vl\") pod \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " Nov 02 17:59:57.682333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.681899 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tls-certs\" (UniqueName: \"kubernetes.io/secret/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tls-certs\") pod \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " Nov 02 17:59:57.682333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.681928 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-home\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-home\") pod \"784056a6-b287-4aa3-af0e-b1175a88200b\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " Nov 02 17:59:57.682333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.681956 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-tmp-dir" pod "784056a6-b287-4aa3-af0e-b1175a88200b" (UID: "784056a6-b287-4aa3-af0e-b1175a88200b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.682333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.681977 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-model-cache\") pod \"784056a6-b287-4aa3-af0e-b1175a88200b\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " Nov 02 17:59:57.682333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682045 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tmp-dir\") pod \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " Nov 02 17:59:57.682333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682095 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-dshm\") pod \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\" (UID: \"72bef2f9-5e9f-44ce-b722-4a09b9d55ee8\") " Nov 02 17:59:57.682333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682107 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-home" pod "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" (UID: "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.682333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682121 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-model-cache" pod "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" (UID: "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.682333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682150 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-dshm\") pod \"784056a6-b287-4aa3-af0e-b1175a88200b\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " Nov 02 17:59:57.682333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682177 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784056a6-b287-4aa3-af0e-b1175a88200b-kube-api-access-9ghx8\" (UniqueName: \"kubernetes.io/projected/784056a6-b287-4aa3-af0e-b1175a88200b-kube-api-access-9ghx8\") pod \"784056a6-b287-4aa3-af0e-b1175a88200b\" (UID: \"784056a6-b287-4aa3-af0e-b1175a88200b\") " Nov 02 17:59:57.682333 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682305 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-model-cache" pod "784056a6-b287-4aa3-af0e-b1175a88200b" (UID: "784056a6-b287-4aa3-af0e-b1175a88200b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.684623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682649 4310 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-tmp-dir\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.684623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682666 4310 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-home\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.684623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682678 4310 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-model-cache\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.684623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682691 4310 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-model-cache\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.684623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.682910 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tmp-dir" pod "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" (UID: "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.684623 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.683208 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-home" pod "784056a6-b287-4aa3-af0e-b1175a88200b" (UID: "784056a6-b287-4aa3-af0e-b1175a88200b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.685192 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.685156 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tls-certs" pod "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" (UID: "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:59:57.685303 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.685250 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784056a6-b287-4aa3-af0e-b1175a88200b-tls-certs" pod "784056a6-b287-4aa3-af0e-b1175a88200b" (UID: "784056a6-b287-4aa3-af0e-b1175a88200b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:59:57.685365 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.685302 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-kube-api-access-rt2vl" pod "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" (UID: "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8"). InnerVolumeSpecName "kube-api-access-rt2vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:57.685414 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.685389 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-dshm" pod "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" (UID: "72bef2f9-5e9f-44ce-b722-4a09b9d55ee8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.685549 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.685534 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784056a6-b287-4aa3-af0e-b1175a88200b-kube-api-access-9ghx8" pod "784056a6-b287-4aa3-af0e-b1175a88200b" (UID: "784056a6-b287-4aa3-af0e-b1175a88200b"). InnerVolumeSpecName "kube-api-access-9ghx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:57.685606 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.685571 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-dshm" pod "784056a6-b287-4aa3-af0e-b1175a88200b" (UID: "784056a6-b287-4aa3-af0e-b1175a88200b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:57.783885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.783809 4310 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784056a6-b287-4aa3-af0e-b1175a88200b-tls-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.783885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.783838 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt2vl\" (UniqueName: \"kubernetes.io/projected/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-kube-api-access-rt2vl\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.783885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.783848 4310 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tls-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.783885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.783856 4310 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-home\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.783885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.783883 4310 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-tmp-dir\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.783885 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.783890 4310 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8-dshm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.784236 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.783899 4310 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784056a6-b287-4aa3-af0e-b1175a88200b-dshm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.784236 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.783912 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ghx8\" (UniqueName: \"kubernetes.io/projected/784056a6-b287-4aa3-af0e-b1175a88200b-kube-api-access-9ghx8\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 17:59:57.911756 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.911724 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz"] Nov 02 17:59:57.915177 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.915148 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-2-w4-ea752f-kserve-7649b5cc6-zw2sz"] Nov 02 17:59:57.926471 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.926444 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz"] Nov 02 17:59:57.929739 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:57.929715 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ea752f/distinct-w4-ea752f-kserve-66d58667cf-kjlmz"] Nov 02 17:59:58.174535 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:58.174434 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:58.674641 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:58.674600 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" path="/var/lib/kubelet/pods/72bef2f9-5e9f-44ce-b722-4a09b9d55ee8/volumes" Nov 02 17:59:58.675229 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:58.675213 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" path="/var/lib/kubelet/pods/784056a6-b287-4aa3-af0e-b1175a88200b/volumes" Nov 02 17:59:58.885283 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:58.885251 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 17:59:59.591593 ip-10-0-135-9 kubenswrapper[4310]: I1102 17:59:59.591552 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:00.166540 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.166485 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8"] Nov 02 18:00:00.167751 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.167722 4310 cpu_manager.go:414] "containerMap: removing container" podUID="7f6b4ee4-b67f-4c91-9db2-103f7c00c052" containerName="revoke-keys" Nov 02 18:00:00.168155 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.168104 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f6b4ee4-b67f-4c91-9db2-103f7c00c052" containerName="revoke-keys" Nov 02 18:00:00.168155 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.168139 4310 cpu_manager.go:414] "containerMap: removing container" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" Nov 02 18:00:00.168155 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.168148 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" Nov 02 18:00:00.168408 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.168161 4310 cpu_manager.go:414] "containerMap: removing container" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" Nov 02 18:00:00.168408 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.168168 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" Nov 02 18:00:00.168408 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.168221 4310 cpu_manager.go:414] "containerMap: removing container" podUID="fa58c510-6474-462b-9455-d676d33e6784" containerName="revoke-keys" Nov 02 18:00:00.168408 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.168228 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa58c510-6474-462b-9455-d676d33e6784" containerName="revoke-keys" Nov 02 18:00:00.168408 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.168371 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa58c510-6474-462b-9455-d676d33e6784" containerName="revoke-keys" Nov 02 18:00:00.168408 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.168389 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f6b4ee4-b67f-4c91-9db2-103f7c00c052" containerName="revoke-keys" Nov 02 18:00:00.168408 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.168403 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="784056a6-b287-4aa3-af0e-b1175a88200b" containerName="main" Nov 02 18:00:00.168850 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.168411 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bef2f9-5e9f-44ce-b722-4a09b9d55ee8" containerName="main" Nov 02 18:00:00.173321 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.173297 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6"] Nov 02 18:00:00.173426 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.173414 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" Nov 02 18:00:00.178344 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.178305 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4"] Nov 02 18:00:00.178485 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.178433 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" Nov 02 18:00:00.180011 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.179990 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:00:00.182619 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.182537 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" Nov 02 18:00:00.183172 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.183148 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8"] Nov 02 18:00:00.204699 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.200938 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6"] Nov 02 18:00:00.205367 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.205336 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4"] Nov 02 18:00:00.311784 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.311737 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv9g5\" (UniqueName: \"kubernetes.io/projected/52879a72-ff92-4f16-83bd-0f65dd98019d-kube-api-access-jv9g5\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4\" (UID: \"52879a72-ff92-4f16-83bd-0f65dd98019d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" Nov 02 18:00:00.311943 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.311862 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpq8h\" (UniqueName: \"kubernetes.io/projected/d4bed6e2-a896-49b5-ad01-4b07d3861161-kube-api-access-fpq8h\") pod \"maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6\" (UID: \"d4bed6e2-a896-49b5-ad01-4b07d3861161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" Nov 02 18:00:00.311943 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.311903 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bed6e2-a896-49b5-ad01-4b07d3861161-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6\" (UID: \"d4bed6e2-a896-49b5-ad01-4b07d3861161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" Nov 02 18:00:00.311943 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.311925 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52879a72-ff92-4f16-83bd-0f65dd98019d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4\" (UID: \"52879a72-ff92-4f16-83bd-0f65dd98019d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" Nov 02 18:00:00.312125 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.312062 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c2061c-374e-4200-b30c-694f64cfb127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8\" (UID: \"a4c2061c-374e-4200-b30c-694f64cfb127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" Nov 02 18:00:00.312201 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.312185 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfllp\" (UniqueName: \"kubernetes.io/projected/a4c2061c-374e-4200-b30c-694f64cfb127-kube-api-access-mfllp\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8\" (UID: \"a4c2061c-374e-4200-b30c-694f64cfb127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" Nov 02 18:00:00.413750 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.413703 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpq8h\" (UniqueName: \"kubernetes.io/projected/d4bed6e2-a896-49b5-ad01-4b07d3861161-kube-api-access-fpq8h\") pod \"maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6\" (UID: \"d4bed6e2-a896-49b5-ad01-4b07d3861161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" Nov 02 18:00:00.413971 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.413761 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bed6e2-a896-49b5-ad01-4b07d3861161-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6\" (UID: \"d4bed6e2-a896-49b5-ad01-4b07d3861161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" Nov 02 18:00:00.413971 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.413788 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52879a72-ff92-4f16-83bd-0f65dd98019d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4\" (UID: \"52879a72-ff92-4f16-83bd-0f65dd98019d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" Nov 02 18:00:00.413971 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.413870 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c2061c-374e-4200-b30c-694f64cfb127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8\" (UID: \"a4c2061c-374e-4200-b30c-694f64cfb127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" Nov 02 18:00:00.413971 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.413930 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfllp\" (UniqueName: \"kubernetes.io/projected/a4c2061c-374e-4200-b30c-694f64cfb127-kube-api-access-mfllp\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8\" (UID: \"a4c2061c-374e-4200-b30c-694f64cfb127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" Nov 02 18:00:00.414250 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.413995 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv9g5\" (UniqueName: \"kubernetes.io/projected/52879a72-ff92-4f16-83bd-0f65dd98019d-kube-api-access-jv9g5\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4\" (UID: \"52879a72-ff92-4f16-83bd-0f65dd98019d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" Nov 02 18:00:00.414570 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.414545 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bed6e2-a896-49b5-ad01-4b07d3861161-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6\" (UID: \"d4bed6e2-a896-49b5-ad01-4b07d3861161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" Nov 02 18:00:00.414656 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.414570 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52879a72-ff92-4f16-83bd-0f65dd98019d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4\" (UID: \"52879a72-ff92-4f16-83bd-0f65dd98019d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" Nov 02 18:00:00.414831 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.414811 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c2061c-374e-4200-b30c-694f64cfb127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8\" (UID: \"a4c2061c-374e-4200-b30c-694f64cfb127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" Nov 02 18:00:00.423899 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.423823 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv9g5\" (UniqueName: \"kubernetes.io/projected/52879a72-ff92-4f16-83bd-0f65dd98019d-kube-api-access-jv9g5\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4\" (UID: \"52879a72-ff92-4f16-83bd-0f65dd98019d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" Nov 02 18:00:00.423899 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.423848 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpq8h\" (UniqueName: \"kubernetes.io/projected/d4bed6e2-a896-49b5-ad01-4b07d3861161-kube-api-access-fpq8h\") pod \"maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6\" (UID: \"d4bed6e2-a896-49b5-ad01-4b07d3861161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" Nov 02 18:00:00.423899 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.423861 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfllp\" (UniqueName: \"kubernetes.io/projected/a4c2061c-374e-4200-b30c-694f64cfb127-kube-api-access-mfllp\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8\" (UID: \"a4c2061c-374e-4200-b30c-694f64cfb127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" Nov 02 18:00:00.468281 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.468242 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:00.494241 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.494203 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" Nov 02 18:00:00.525107 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.524317 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" Nov 02 18:00:00.546049 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.526224 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" Nov 02 18:00:00.840799 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.840766 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8"] Nov 02 18:00:00.841236 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:00:00.841210 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4bed6e2_a896_49b5_ad01_4b07d3861161.slice/crio-a400f904a47bccb1cb0e64be7dc2ccbbebe79057d8a5141b7648dea66da3a2ad WatchSource:0}: Error finding container a400f904a47bccb1cb0e64be7dc2ccbbebe79057d8a5141b7648dea66da3a2ad: Status 404 returned error can't find the container with id a400f904a47bccb1cb0e64be7dc2ccbbebe79057d8a5141b7648dea66da3a2ad Nov 02 18:00:00.841686 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:00:00.841638 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c2061c_374e_4200_b30c_694f64cfb127.slice/crio-309f5c4ad4d9bff2aa7056f6fb9f47283dd22b2222d6dbdaba74fae9d15162ae WatchSource:0}: Error finding container 309f5c4ad4d9bff2aa7056f6fb9f47283dd22b2222d6dbdaba74fae9d15162ae: Status 404 returned error can't find the container with id 309f5c4ad4d9bff2aa7056f6fb9f47283dd22b2222d6dbdaba74fae9d15162ae Nov 02 18:00:00.844505 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.844476 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6"] Nov 02 18:00:00.851414 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:00.851386 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4"] Nov 02 18:00:00.854950 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:00:00.854921 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52879a72_ff92_4f16_83bd_0f65dd98019d.slice/crio-aa15a534ee1cdcf9eaf086d855e6089b3994af7605c3319efa145d5236ff3360 WatchSource:0}: Error finding container aa15a534ee1cdcf9eaf086d855e6089b3994af7605c3319efa145d5236ff3360: Status 404 returned error can't find the container with id aa15a534ee1cdcf9eaf086d855e6089b3994af7605c3319efa145d5236ff3360 Nov 02 18:00:01.176337 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.176304 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:01.217569 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.217529 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" podUID="12043acc-60b0-48d1-9164-127fc5215f28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:00:01.658593 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.658557 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" event={"ID":"a4c2061c-374e-4200-b30c-694f64cfb127","Type":"ContainerStarted","Data":"91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365"} Nov 02 18:00:01.658758 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.658612 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" event={"ID":"a4c2061c-374e-4200-b30c-694f64cfb127","Type":"ContainerStarted","Data":"309f5c4ad4d9bff2aa7056f6fb9f47283dd22b2222d6dbdaba74fae9d15162ae"} Nov 02 18:00:01.660405 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.660375 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" event={"ID":"52879a72-ff92-4f16-83bd-0f65dd98019d","Type":"ContainerStarted","Data":"781e883d343a26ae27497a1e1a29b822b0634eeab8bbb960c2145fff6348edfc"} Nov 02 18:00:01.660545 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.660426 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" event={"ID":"52879a72-ff92-4f16-83bd-0f65dd98019d","Type":"ContainerStarted","Data":"aa15a534ee1cdcf9eaf086d855e6089b3994af7605c3319efa145d5236ff3360"} Nov 02 18:00:01.662424 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.662392 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" event={"ID":"d4bed6e2-a896-49b5-ad01-4b07d3861161","Type":"ContainerStarted","Data":"220e6473a0e8c541aea4cda0cc2bdad450061da31446ee17eeaa850a611bfb8e"} Nov 02 18:00:01.662557 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.662442 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" event={"ID":"d4bed6e2-a896-49b5-ad01-4b07d3861161","Type":"ContainerStarted","Data":"a400f904a47bccb1cb0e64be7dc2ccbbebe79057d8a5141b7648dea66da3a2ad"} Nov 02 18:00:01.698787 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.698735 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" podStartSLOduration=1.6987226039999999 podStartE2EDuration="1.698722604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:01.696233501 +0000 UTC m=+1199.139316138" watchObservedRunningTime="2026-11-02 18:00:01.698722604 +0000 UTC m=+1199.141805240" Nov 02 18:00:01.700005 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.699963 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" podStartSLOduration=1.699950103 podStartE2EDuration="1.699950103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:01.677988364 +0000 UTC m=+1199.121071000" watchObservedRunningTime="2026-11-02 18:00:01.699950103 +0000 UTC m=+1199.143032739" Nov 02 18:00:01.713588 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.713471 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" podStartSLOduration=1.713452083 podStartE2EDuration="1.713452083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:01.712849138 +0000 UTC m=+1199.155931774" watchObservedRunningTime="2026-11-02 18:00:01.713452083 +0000 UTC m=+1199.156534725" Nov 02 18:00:01.875848 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:01.875824 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:02.725039 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:02.724998 4310 generic.go:356] "Generic (PLEG): container finished" podID="a4c2061c-374e-4200-b30c-694f64cfb127" containerID="91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365" exitCode=60 Nov 02 18:00:02.725900 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:02.725871 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" event={"ID":"a4c2061c-374e-4200-b30c-694f64cfb127","Type":"ContainerDied","Data":"91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365"} Nov 02 18:00:02.726573 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:02.726552 4310 scope.go:122] "RemoveContainer" containerID="91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365" Nov 02 18:00:02.741062 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:02.741025 4310 generic.go:356] "Generic (PLEG): container finished" podID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerID="781e883d343a26ae27497a1e1a29b822b0634eeab8bbb960c2145fff6348edfc" exitCode=60 Nov 02 18:00:02.741218 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:02.741196 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" event={"ID":"52879a72-ff92-4f16-83bd-0f65dd98019d","Type":"ContainerDied","Data":"781e883d343a26ae27497a1e1a29b822b0634eeab8bbb960c2145fff6348edfc"} Nov 02 18:00:02.741731 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:02.741709 4310 scope.go:122] "RemoveContainer" containerID="781e883d343a26ae27497a1e1a29b822b0634eeab8bbb960c2145fff6348edfc" Nov 02 18:00:02.771992 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:02.771872 4310 generic.go:356] "Generic (PLEG): container finished" podID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerID="220e6473a0e8c541aea4cda0cc2bdad450061da31446ee17eeaa850a611bfb8e" exitCode=60 Nov 02 18:00:02.772181 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:02.772032 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" event={"ID":"d4bed6e2-a896-49b5-ad01-4b07d3861161","Type":"ContainerDied","Data":"220e6473a0e8c541aea4cda0cc2bdad450061da31446ee17eeaa850a611bfb8e"} Nov 02 18:00:02.772545 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:02.772520 4310 scope.go:122] "RemoveContainer" containerID="220e6473a0e8c541aea4cda0cc2bdad450061da31446ee17eeaa850a611bfb8e" Nov 02 18:00:02.826019 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:02.825981 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:03.655545 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:03.655455 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8"] Nov 02 18:00:03.737026 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:03.736989 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:03.800131 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:03.787377 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" event={"ID":"52879a72-ff92-4f16-83bd-0f65dd98019d","Type":"ContainerStarted","Data":"bb9810270ff02e5e24e6f70dfea86c612eb6b479d787595a94fd2ed92fac7223"} Nov 02 18:00:03.800131 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:03.791925 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" event={"ID":"d4bed6e2-a896-49b5-ad01-4b07d3861161","Type":"ContainerStarted","Data":"f2f08352dd40c33b01a1c393fc0d3c142c3ea3d396530d378ac585984ecdb092"} Nov 02 18:00:03.800131 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:03.797134 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" event={"ID":"a4c2061c-374e-4200-b30c-694f64cfb127","Type":"ContainerStarted","Data":"6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4"} Nov 02 18:00:04.051098 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.049485 4310 scope.go:122] "RemoveContainer" containerID="dcae7957dcc1cf3630fa2df988b7853347f07f6dc54cd4e63864673803eeaaf1" Nov 02 18:00:04.072364 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:04.072306 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:00:04.075264 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.075233 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:04.262101 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.243992 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:04.312635 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.312449 4310 scope.go:122] "RemoveContainer" containerID="964990334dfb586323be9a359f2be7a2e4e86191058cc5d678098a2f9067eec0" Nov 02 18:00:04.364870 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.363847 4310 scope.go:122] "RemoveContainer" containerID="148c3b34ea16b536aa7bf837cb880cff23594109a1ec226cddb54a8d85174aff" Nov 02 18:00:04.367301 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.367274 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv"] Nov 02 18:00:04.373376 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.373348 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea752f-rh6fv"] Nov 02 18:00:04.379278 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.379252 4310 scope.go:122] "RemoveContainer" containerID="4fbf1f0d8b35beff7efb98441e0338b33b239d3e88c3d25da69c750ab23a1a8f" Nov 02 18:00:04.391169 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.391144 4310 scope.go:122] "RemoveContainer" containerID="b513e8dafdff4b6efe2ef9a9d4a9256992110379ee4dde01df9cc5ad72f25020" Nov 02 18:00:04.401011 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.400986 4310 scope.go:122] "RemoveContainer" containerID="7ae82469085929472726d10c20baf10affcd0edcef982610bf97634aab322c92" Nov 02 18:00:04.410698 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.410678 4310 scope.go:122] "RemoveContainer" containerID="9ee9e88dcae8e17ad8b849535462abd74df7babfb243349b1abeda848fa8f2f2" Nov 02 18:00:04.420558 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.420541 4310 scope.go:122] "RemoveContainer" containerID="a9ce481d303ddf83b00824f47e556499dc4fa7d4139f11f52037275d7ed39852" Nov 02 18:00:04.429467 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.429444 4310 scope.go:122] "RemoveContainer" containerID="1d1100c9358a7e40b36c15c93483bae9ed77ddf3b72a8b6dfd2231ad16d0971e" Nov 02 18:00:04.438698 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.438677 4310 scope.go:122] "RemoveContainer" containerID="eccab6c439029c57d1a61d1d86335b40c67a1f23795340aebf39eb0c5a929ff4" Nov 02 18:00:04.449021 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.449004 4310 scope.go:122] "RemoveContainer" containerID="2208095a031039f93d088092e965bad278b28bf96423c1ccdf82102f92df6a29" Nov 02 18:00:04.457981 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.457964 4310 scope.go:122] "RemoveContainer" containerID="b836eac27b9f00fcc13bc6fb659fbda1bca4b11b918102dad54d20c9634838ea" Nov 02 18:00:04.569638 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.569595 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:04.676892 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.676861 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2878b8cb-1876-43e5-902d-a4dfd302ef8f" path="/var/lib/kubelet/pods/2878b8cb-1876-43e5-902d-a4dfd302ef8f/volumes" Nov 02 18:00:04.771117 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.771051 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:04.805948 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.805910 4310 generic.go:356] "Generic (PLEG): container finished" podID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerID="bb9810270ff02e5e24e6f70dfea86c612eb6b479d787595a94fd2ed92fac7223" exitCode=60 Nov 02 18:00:04.806140 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.805942 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" event={"ID":"52879a72-ff92-4f16-83bd-0f65dd98019d","Type":"ContainerDied","Data":"bb9810270ff02e5e24e6f70dfea86c612eb6b479d787595a94fd2ed92fac7223"} Nov 02 18:00:04.806140 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.806019 4310 scope.go:122] "RemoveContainer" containerID="781e883d343a26ae27497a1e1a29b822b0634eeab8bbb960c2145fff6348edfc" Nov 02 18:00:04.806412 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.806323 4310 scope.go:122] "RemoveContainer" containerID="bb9810270ff02e5e24e6f70dfea86c612eb6b479d787595a94fd2ed92fac7223" Nov 02 18:00:04.806684 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:04.806656 4310 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4_odh-ai-gateway-infra(52879a72-ff92-4f16-83bd-0f65dd98019d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" Nov 02 18:00:04.808344 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.808323 4310 generic.go:356] "Generic (PLEG): container finished" podID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerID="f2f08352dd40c33b01a1c393fc0d3c142c3ea3d396530d378ac585984ecdb092" exitCode=60 Nov 02 18:00:04.808451 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.808399 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" event={"ID":"d4bed6e2-a896-49b5-ad01-4b07d3861161","Type":"ContainerDied","Data":"f2f08352dd40c33b01a1c393fc0d3c142c3ea3d396530d378ac585984ecdb092"} Nov 02 18:00:04.808614 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.808593 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" podUID="a4c2061c-374e-4200-b30c-694f64cfb127" containerName="cleanup" containerID="cri-o://6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4" gracePeriod=30 Nov 02 18:00:04.808667 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.808617 4310 scope.go:122] "RemoveContainer" containerID="f2f08352dd40c33b01a1c393fc0d3c142c3ea3d396530d378ac585984ecdb092" Nov 02 18:00:04.808984 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:04.808863 4310 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6_odh-ai-gateway-infra(d4bed6e2-a896-49b5-ad01-4b07d3861161)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" Nov 02 18:00:04.828688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:04.828558 4310 scope.go:122] "RemoveContainer" containerID="220e6473a0e8c541aea4cda0cc2bdad450061da31446ee17eeaa850a611bfb8e" Nov 02 18:00:05.025309 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:05.025275 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w"] Nov 02 18:00:05.030570 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:05.030540 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c75a54-wdn2w"] Nov 02 18:00:05.488819 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:05.488779 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:05.817306 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:05.817227 4310 scope.go:122] "RemoveContainer" containerID="f2f08352dd40c33b01a1c393fc0d3c142c3ea3d396530d378ac585984ecdb092" Nov 02 18:00:05.817713 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:05.817523 4310 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6_odh-ai-gateway-infra(d4bed6e2-a896-49b5-ad01-4b07d3861161)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" Nov 02 18:00:05.819129 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:05.819107 4310 scope.go:122] "RemoveContainer" containerID="bb9810270ff02e5e24e6f70dfea86c612eb6b479d787595a94fd2ed92fac7223" Nov 02 18:00:05.819360 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:05.819338 4310 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4_odh-ai-gateway-infra(52879a72-ff92-4f16-83bd-0f65dd98019d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" Nov 02 18:00:06.532072 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:06.532034 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:06.674434 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:06.674403 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc7d6535-bc17-41ad-bd97-87013fa13171" path="/var/lib/kubelet/pods/cc7d6535-bc17-41ad-bd97-87013fa13171/volumes" Nov 02 18:00:07.200338 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:07.200305 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:07.872579 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:07.872551 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:08.691389 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:08.691358 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:09.493374 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:09.493338 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:10.192129 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:10.192068 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:11.057558 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:11.057523 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:11.216833 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:11.216801 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" podUID="12043acc-60b0-48d1-9164-127fc5215f28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:00:11.776723 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:11.776689 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:12.100613 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.100528 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:12.223888 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.223856 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:12.469530 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.469484 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm"] Nov 02 18:00:12.469802 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.469775 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" containerID="cri-o://a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a" gracePeriod=30 Nov 02 18:00:12.474817 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.474793 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb"] Nov 02 18:00:12.475262 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.475230 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" containerID="cri-o://ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e" gracePeriod=30 Nov 02 18:00:12.478550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.478530 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj"] Nov 02 18:00:12.478757 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.478734 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" containerID="cri-o://87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf" gracePeriod=30 Nov 02 18:00:12.486297 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.486273 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k"] Nov 02 18:00:12.486465 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.486447 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" containerID="cri-o://fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773" gracePeriod=30 Nov 02 18:00:12.511235 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.511207 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:12.785877 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.785841 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 18:00:12.808103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.804147 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:12.885545 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.885513 4310 generic.go:356] "Generic (PLEG): container finished" podID="12043acc-60b0-48d1-9164-127fc5215f28" containerID="ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57" exitCode=0 Nov 02 18:00:12.885718 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.885562 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" Nov 02 18:00:12.885718 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.885591 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" event={"ID":"12043acc-60b0-48d1-9164-127fc5215f28","Type":"ContainerDied","Data":"ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57"} Nov 02 18:00:12.885718 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.885652 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l" event={"ID":"12043acc-60b0-48d1-9164-127fc5215f28","Type":"ContainerDied","Data":"aa717aa1b03da112140be268718989271277ce18fcf978c93b54d1e2140e1907"} Nov 02 18:00:12.885718 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.885707 4310 scope.go:122] "RemoveContainer" containerID="ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57" Nov 02 18:00:12.896037 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.896015 4310 scope.go:122] "RemoveContainer" containerID="ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57" Nov 02 18:00:12.896467 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:12.896443 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57\": container with ID starting with ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57 not found: ID does not exist" containerID="ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57" Nov 02 18:00:12.896587 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.896471 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57"} err="failed to get container status \"ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57\": rpc error: code = NotFound desc = could not find container \"ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57\": container with ID starting with ce3b46b342ce75d2829a87d44727d8493b9d68e3208ff0b5059e03972d0aee57 not found: ID does not exist" Nov 02 18:00:12.917967 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.917939 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12043acc-60b0-48d1-9164-127fc5215f28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12043acc-60b0-48d1-9164-127fc5215f28-plugins-config-volume\") pod \"12043acc-60b0-48d1-9164-127fc5215f28\" (UID: \"12043acc-60b0-48d1-9164-127fc5215f28\") " Nov 02 18:00:12.918113 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.918001 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12043acc-60b0-48d1-9164-127fc5215f28-kube-api-access-t86pt\" (UniqueName: \"kubernetes.io/projected/12043acc-60b0-48d1-9164-127fc5215f28-kube-api-access-t86pt\") pod \"12043acc-60b0-48d1-9164-127fc5215f28\" (UID: \"12043acc-60b0-48d1-9164-127fc5215f28\") " Nov 02 18:00:12.918349 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.918330 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12043acc-60b0-48d1-9164-127fc5215f28-plugins-config-volume" pod "12043acc-60b0-48d1-9164-127fc5215f28" (UID: "12043acc-60b0-48d1-9164-127fc5215f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:12.918495 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.918482 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12043acc-60b0-48d1-9164-127fc5215f28-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:12.920048 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:12.920029 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12043acc-60b0-48d1-9164-127fc5215f28-kube-api-access-t86pt" pod "12043acc-60b0-48d1-9164-127fc5215f28" (UID: "12043acc-60b0-48d1-9164-127fc5215f28"). InnerVolumeSpecName "kube-api-access-t86pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:13.019753 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:13.019674 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t86pt\" (UniqueName: \"kubernetes.io/projected/12043acc-60b0-48d1-9164-127fc5215f28-kube-api-access-t86pt\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:13.214865 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:13.214757 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l"] Nov 02 18:00:13.224751 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:13.224719 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea752f-6b697785ff-5sq8l"] Nov 02 18:00:13.487924 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:13.487881 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:14.290391 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:14.290343 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:14.675739 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:14.675652 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12043acc-60b0-48d1-9164-127fc5215f28" path="/var/lib/kubelet/pods/12043acc-60b0-48d1-9164-127fc5215f28/volumes" Nov 02 18:00:15.028674 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:15.023891 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:15.738006 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:15.737972 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:15.792435 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:15.792382 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:00:15.792798 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:15.792770 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:00:16.033005 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:16.032917 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:00:16.033005 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:16.032981 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:00:16.426886 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:16.426788 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:16.674065 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:16.674037 4310 scope.go:122] "RemoveContainer" containerID="f2f08352dd40c33b01a1c393fc0d3c142c3ea3d396530d378ac585984ecdb092" Nov 02 18:00:17.193736 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.193691 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:17.424493 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.424411 4310 patch_prober.go:29] interesting pod/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:00:17.424493 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.424477 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:00:17.674042 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.673901 4310 scope.go:122] "RemoveContainer" containerID="bb9810270ff02e5e24e6f70dfea86c612eb6b479d787595a94fd2ed92fac7223" Nov 02 18:00:17.890260 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.890234 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 18:00:17.894410 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.894386 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 18:00:17.899032 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.899009 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 18:00:17.902137 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.902117 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 18:00:17.925355 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.925327 4310 generic.go:356] "Generic (PLEG): container finished" podID="2d182005-2390-4c46-8327-c051ce27933a" containerID="fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773" exitCode=0 Nov 02 18:00:17.925534 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.925381 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" Nov 02 18:00:17.925534 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.925422 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" event={"ID":"2d182005-2390-4c46-8327-c051ce27933a","Type":"ContainerDied","Data":"fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773"} Nov 02 18:00:17.925534 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.925487 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k" event={"ID":"2d182005-2390-4c46-8327-c051ce27933a","Type":"ContainerDied","Data":"603b77dbef38c86a757476d9db0212d06880c9afb2c9652014507c7a56e22141"} Nov 02 18:00:17.925534 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.925521 4310 scope.go:122] "RemoveContainer" containerID="fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773" Nov 02 18:00:17.927846 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.927825 4310 generic.go:356] "Generic (PLEG): container finished" podID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerID="87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf" exitCode=0 Nov 02 18:00:17.927962 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.927884 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" Nov 02 18:00:17.927962 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.927918 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" event={"ID":"ca5d24c0-427d-4b1c-b706-965be02d81d0","Type":"ContainerDied","Data":"87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf"} Nov 02 18:00:17.928056 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.927997 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj" event={"ID":"ca5d24c0-427d-4b1c-b706-965be02d81d0","Type":"ContainerDied","Data":"456a10cac7878eab7e887c5587c3bf69068cb34804b4585fbb8ba595089731c4"} Nov 02 18:00:17.930212 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.930148 4310 generic.go:356] "Generic (PLEG): container finished" podID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerID="2e0f362e1e653126f7732724be4001247570749c0b6d5aebb46892f3da854129" exitCode=60 Nov 02 18:00:17.930333 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.930232 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" event={"ID":"d4bed6e2-a896-49b5-ad01-4b07d3861161","Type":"ContainerDied","Data":"2e0f362e1e653126f7732724be4001247570749c0b6d5aebb46892f3da854129"} Nov 02 18:00:17.930642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.930619 4310 scope.go:122] "RemoveContainer" containerID="2e0f362e1e653126f7732724be4001247570749c0b6d5aebb46892f3da854129" Nov 02 18:00:17.930981 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:17.930956 4310 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6_odh-ai-gateway-infra(d4bed6e2-a896-49b5-ad01-4b07d3861161)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" Nov 02 18:00:17.932041 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.932013 4310 generic.go:356] "Generic (PLEG): container finished" podID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerID="ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e" exitCode=0 Nov 02 18:00:17.932172 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.932154 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" event={"ID":"3f164162-c2b6-41de-9068-ca6eedf8417f","Type":"ContainerDied","Data":"ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e"} Nov 02 18:00:17.932225 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.932202 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" event={"ID":"3f164162-c2b6-41de-9068-ca6eedf8417f","Type":"ContainerDied","Data":"55714b404783ae5bd22a46b8172571ca541c2bd78a26dc9d3f02d710978064f5"} Nov 02 18:00:17.932279 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.932267 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb" Nov 02 18:00:17.934023 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.934001 4310 generic.go:356] "Generic (PLEG): container finished" podID="ded09c48-50f3-417f-9502-39403c2878d4" containerID="a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a" exitCode=0 Nov 02 18:00:17.934156 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.934031 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" event={"ID":"ded09c48-50f3-417f-9502-39403c2878d4","Type":"ContainerDied","Data":"a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a"} Nov 02 18:00:17.934156 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.934053 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" Nov 02 18:00:17.934156 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.934064 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm" event={"ID":"ded09c48-50f3-417f-9502-39403c2878d4","Type":"ContainerDied","Data":"c0f630756027c9e97b1c3a20ef9413b8518f4b514f48dc716df0a974bc066e2d"} Nov 02 18:00:17.938791 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.938768 4310 scope.go:122] "RemoveContainer" containerID="eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e" Nov 02 18:00:17.953561 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.953537 4310 scope.go:122] "RemoveContainer" containerID="fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773" Nov 02 18:00:17.953953 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:17.953928 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773\": container with ID starting with fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773 not found: ID does not exist" containerID="fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773" Nov 02 18:00:17.954043 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.953968 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773"} err="failed to get container status \"fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773\": rpc error: code = NotFound desc = could not find container \"fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773\": container with ID starting with fdbbb5cd4cd6eb9bebe6e995d28543e9b2ad782e2efbc93ab25e5bcc5828b773 not found: ID does not exist" Nov 02 18:00:17.954043 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.953994 4310 scope.go:122] "RemoveContainer" containerID="eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e" Nov 02 18:00:17.954368 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:17.954349 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e\": container with ID starting with eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e not found: ID does not exist" containerID="eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e" Nov 02 18:00:17.954412 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.954376 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e"} err="failed to get container status \"eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e\": rpc error: code = NotFound desc = could not find container \"eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e\": container with ID starting with eb14182e8a81e5d1cd276328e6a3f50f83ab3de5a35bb567d907c6237578516e not found: ID does not exist" Nov 02 18:00:17.954412 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.954394 4310 scope.go:122] "RemoveContainer" containerID="87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf" Nov 02 18:00:17.965034 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.965012 4310 scope.go:122] "RemoveContainer" containerID="87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf" Nov 02 18:00:17.965405 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:17.965371 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf\": container with ID starting with 87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf not found: ID does not exist" containerID="87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf" Nov 02 18:00:17.965470 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.965411 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf"} err="failed to get container status \"87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf\": rpc error: code = NotFound desc = could not find container \"87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf\": container with ID starting with 87ccd73febb2f7c1121ff9a820001547d5d142f07b955998a28b281f5c257baf not found: ID does not exist" Nov 02 18:00:17.965470 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.965436 4310 scope.go:122] "RemoveContainer" containerID="f2f08352dd40c33b01a1c393fc0d3c142c3ea3d396530d378ac585984ecdb092" Nov 02 18:00:17.983116 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.983071 4310 scope.go:122] "RemoveContainer" containerID="ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e" Nov 02 18:00:17.994948 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.994925 4310 scope.go:122] "RemoveContainer" containerID="ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e" Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995418 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-credential-socket\") pod \"3f164162-c2b6-41de-9068-ca6eedf8417f\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:17.995411 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e\": container with ID starting with ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e not found: ID does not exist" containerID="ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e" Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995479 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-kube-api-access-76m8b\" (UniqueName: \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-kube-api-access-76m8b\") pod \"3f164162-c2b6-41de-9068-ca6eedf8417f\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995491 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e"} err="failed to get container status \"ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e\": rpc error: code = NotFound desc = could not find container \"ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e\": container with ID starting with ea2c36cf8e873a39bd08512208c30d23d852176ca955384e3034e8cc4ded383e not found: ID does not exist" Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995509 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-certs\") pod \"ded09c48-50f3-417f-9502-39403c2878d4\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995527 4310 scope.go:122] "RemoveContainer" containerID="a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a" Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995546 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-envoy\") pod \"2d182005-2390-4c46-8327-c051ce27933a\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995573 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-podinfo\") pod \"ca5d24c0-427d-4b1c-b706-965be02d81d0\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995601 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-credential-socket\") pod \"ca5d24c0-427d-4b1c-b706-965be02d81d0\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995633 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-socket\") pod \"3f164162-c2b6-41de-9068-ca6eedf8417f\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995659 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ded09c48-50f3-417f-9502-39403c2878d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded09c48-50f3-417f-9502-39403c2878d4-istio-podinfo\") pod \"ded09c48-50f3-417f-9502-39403c2878d4\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995669 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-credential-socket" pod "3f164162-c2b6-41de-9068-ca6eedf8417f" (UID: "3f164162-c2b6-41de-9068-ca6eedf8417f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995698 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d182005-2390-4c46-8327-c051ce27933a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d182005-2390-4c46-8327-c051ce27933a-istiod-ca-cert\") pod \"2d182005-2390-4c46-8327-c051ce27933a\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995724 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-kube-api-access-ts9bv\" (UniqueName: \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-kube-api-access-ts9bv\") pod \"ded09c48-50f3-417f-9502-39403c2878d4\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995753 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-credential-socket\") pod \"ded09c48-50f3-417f-9502-39403c2878d4\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995783 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-envoy\") pod \"ded09c48-50f3-417f-9502-39403c2878d4\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995818 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-podinfo\") pod \"3f164162-c2b6-41de-9068-ca6eedf8417f\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995853 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-credential-socket\") pod \"2d182005-2390-4c46-8327-c051ce27933a\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995882 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-envoy\") pod \"ca5d24c0-427d-4b1c-b706-965be02d81d0\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.995969 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-istio-token\" (UniqueName: \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-istio-token\") pod \"ded09c48-50f3-417f-9502-39403c2878d4\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996033 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-data\") pod \"3f164162-c2b6-41de-9068-ca6eedf8417f\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996068 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-data\") pod \"ded09c48-50f3-417f-9502-39403c2878d4\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996115 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-socket\") pod \"ded09c48-50f3-417f-9502-39403c2878d4\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996151 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-certs\") pod \"ca5d24c0-427d-4b1c-b706-965be02d81d0\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996185 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f164162-c2b6-41de-9068-ca6eedf8417f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f164162-c2b6-41de-9068-ca6eedf8417f-istiod-ca-cert\") pod \"3f164162-c2b6-41de-9068-ca6eedf8417f\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996270 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-kube-api-access-4svw9\" (UniqueName: \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-kube-api-access-4svw9\") pod \"2d182005-2390-4c46-8327-c051ce27933a\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996305 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-data\") pod \"2d182005-2390-4c46-8327-c051ce27933a\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996334 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5d24c0-427d-4b1c-b706-965be02d81d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5d24c0-427d-4b1c-b706-965be02d81d0-istiod-ca-cert\") pod \"ca5d24c0-427d-4b1c-b706-965be02d81d0\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996370 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-socket" pod "3f164162-c2b6-41de-9068-ca6eedf8417f" (UID: "3f164162-c2b6-41de-9068-ca6eedf8417f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996381 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-istio-token\" (UniqueName: \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-istio-token\") pod \"2d182005-2390-4c46-8327-c051ce27933a\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996470 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-token\" (UniqueName: \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-token\") pod \"3f164162-c2b6-41de-9068-ca6eedf8417f\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996551 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-certs\") pod \"3f164162-c2b6-41de-9068-ca6eedf8417f\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996610 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-certs\") pod \"2d182005-2390-4c46-8327-c051ce27933a\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996671 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d182005-2390-4c46-8327-c051ce27933a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d182005-2390-4c46-8327-c051ce27933a-istio-podinfo\") pod \"2d182005-2390-4c46-8327-c051ce27933a\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996707 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-token\" (UniqueName: \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-token\") pod \"ca5d24c0-427d-4b1c-b706-965be02d81d0\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " Nov 02 18:00:17.996759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996760 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-socket\") pod \"2d182005-2390-4c46-8327-c051ce27933a\" (UID: \"2d182005-2390-4c46-8327-c051ce27933a\") " Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996788 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded09c48-50f3-417f-9502-39403c2878d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded09c48-50f3-417f-9502-39403c2878d4-istiod-ca-cert\") pod \"ded09c48-50f3-417f-9502-39403c2878d4\" (UID: \"ded09c48-50f3-417f-9502-39403c2878d4\") " Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996863 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-socket\") pod \"ca5d24c0-427d-4b1c-b706-965be02d81d0\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996930 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-envoy\") pod \"3f164162-c2b6-41de-9068-ca6eedf8417f\" (UID: \"3f164162-c2b6-41de-9068-ca6eedf8417f\") " Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.996994 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-data\") pod \"ca5d24c0-427d-4b1c-b706-965be02d81d0\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.997057 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-credential-socket" pod "ca5d24c0-427d-4b1c-b706-965be02d81d0" (UID: "ca5d24c0-427d-4b1c-b706-965be02d81d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.997124 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-kube-api-access-v7ckk\" (UniqueName: \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-kube-api-access-v7ckk\") pod \"ca5d24c0-427d-4b1c-b706-965be02d81d0\" (UID: \"ca5d24c0-427d-4b1c-b706-965be02d81d0\") " Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.997484 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-data" pod "2d182005-2390-4c46-8327-c051ce27933a" (UID: "2d182005-2390-4c46-8327-c051ce27933a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.997576 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-certs" pod "ded09c48-50f3-417f-9502-39403c2878d4" (UID: "ded09c48-50f3-417f-9502-39403c2878d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.997754 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-credential-socket" pod "ded09c48-50f3-417f-9502-39403c2878d4" (UID: "ded09c48-50f3-417f-9502-39403c2878d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.997955 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5d24c0-427d-4b1c-b706-965be02d81d0-istiod-ca-cert" pod "ca5d24c0-427d-4b1c-b706-965be02d81d0" (UID: "ca5d24c0-427d-4b1c-b706-965be02d81d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.998221 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.998249 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.998264 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.998280 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.998294 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.998309 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:17.998918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.998323 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5d24c0-427d-4b1c-b706-965be02d81d0-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.000031 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:17.999242 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-socket" pod "2d182005-2390-4c46-8327-c051ce27933a" (UID: "2d182005-2390-4c46-8327-c051ce27933a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.000215 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.000190 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-certs" pod "3f164162-c2b6-41de-9068-ca6eedf8417f" (UID: "3f164162-c2b6-41de-9068-ca6eedf8417f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.000416 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.000396 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-certs" pod "2d182005-2390-4c46-8327-c051ce27933a" (UID: "2d182005-2390-4c46-8327-c051ce27933a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.002553 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.002493 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d182005-2390-4c46-8327-c051ce27933a-istiod-ca-cert" pod "2d182005-2390-4c46-8327-c051ce27933a" (UID: "2d182005-2390-4c46-8327-c051ce27933a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:18.003444 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.003192 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded09c48-50f3-417f-9502-39403c2878d4-istiod-ca-cert" pod "ded09c48-50f3-417f-9502-39403c2878d4" (UID: "ded09c48-50f3-417f-9502-39403c2878d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:18.004118 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.003778 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-data" pod "ca5d24c0-427d-4b1c-b706-965be02d81d0" (UID: "ca5d24c0-427d-4b1c-b706-965be02d81d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.004330 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.004283 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-socket" pod "ca5d24c0-427d-4b1c-b706-965be02d81d0" (UID: "ca5d24c0-427d-4b1c-b706-965be02d81d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.004552 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.004307 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-data" pod "ded09c48-50f3-417f-9502-39403c2878d4" (UID: "ded09c48-50f3-417f-9502-39403c2878d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.004552 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.004495 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-credential-socket" pod "2d182005-2390-4c46-8327-c051ce27933a" (UID: "2d182005-2390-4c46-8327-c051ce27933a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.004936 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.004683 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-data" pod "3f164162-c2b6-41de-9068-ca6eedf8417f" (UID: "3f164162-c2b6-41de-9068-ca6eedf8417f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.005747 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.005450 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f164162-c2b6-41de-9068-ca6eedf8417f-istiod-ca-cert" pod "3f164162-c2b6-41de-9068-ca6eedf8417f" (UID: "3f164162-c2b6-41de-9068-ca6eedf8417f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:18.005747 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.005681 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-certs" pod "ca5d24c0-427d-4b1c-b706-965be02d81d0" (UID: "ca5d24c0-427d-4b1c-b706-965be02d81d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.005942 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.005839 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-socket" pod "ded09c48-50f3-417f-9502-39403c2878d4" (UID: "ded09c48-50f3-417f-9502-39403c2878d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.015582 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.015564 4310 scope.go:122] "RemoveContainer" containerID="a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a" Nov 02 18:00:18.015850 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:18.015832 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a\": container with ID starting with a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a not found: ID does not exist" containerID="a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a" Nov 02 18:00:18.015915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.015857 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a"} err="failed to get container status \"a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a\": rpc error: code = NotFound desc = could not find container \"a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a\": container with ID starting with a322225377e26c49346a836518ab3cbf06001a674efa780bf1002346cbbc180a not found: ID does not exist" Nov 02 18:00:18.043045 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.043007 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-kube-api-access-76m8b" pod "3f164162-c2b6-41de-9068-ca6eedf8417f" (UID: "3f164162-c2b6-41de-9068-ca6eedf8417f"). InnerVolumeSpecName "kube-api-access-76m8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:18.043372 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.043334 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-envoy" pod "ded09c48-50f3-417f-9502-39403c2878d4" (UID: "ded09c48-50f3-417f-9502-39403c2878d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.043482 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.043347 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-envoy" pod "2d182005-2390-4c46-8327-c051ce27933a" (UID: "2d182005-2390-4c46-8327-c051ce27933a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.043482 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.043376 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-token" pod "ca5d24c0-427d-4b1c-b706-965be02d81d0" (UID: "ca5d24c0-427d-4b1c-b706-965be02d81d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:18.043482 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.043368 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-envoy" pod "3f164162-c2b6-41de-9068-ca6eedf8417f" (UID: "3f164162-c2b6-41de-9068-ca6eedf8417f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.043482 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.043404 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-token" pod "3f164162-c2b6-41de-9068-ca6eedf8417f" (UID: "3f164162-c2b6-41de-9068-ca6eedf8417f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:18.043482 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.043417 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d182005-2390-4c46-8327-c051ce27933a-istio-podinfo" pod "2d182005-2390-4c46-8327-c051ce27933a" (UID: "2d182005-2390-4c46-8327-c051ce27933a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:18.043721 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.043668 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-istio-token" pod "2d182005-2390-4c46-8327-c051ce27933a" (UID: "2d182005-2390-4c46-8327-c051ce27933a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:18.043842 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.043819 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-podinfo" pod "3f164162-c2b6-41de-9068-ca6eedf8417f" (UID: "3f164162-c2b6-41de-9068-ca6eedf8417f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:18.043923 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.043872 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-podinfo" pod "ca5d24c0-427d-4b1c-b706-965be02d81d0" (UID: "ca5d24c0-427d-4b1c-b706-965be02d81d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:18.043980 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.043964 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-kube-api-access-4svw9" pod "2d182005-2390-4c46-8327-c051ce27933a" (UID: "2d182005-2390-4c46-8327-c051ce27933a"). InnerVolumeSpecName "kube-api-access-4svw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:18.044132 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.044110 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-kube-api-access-v7ckk" pod "ca5d24c0-427d-4b1c-b706-965be02d81d0" (UID: "ca5d24c0-427d-4b1c-b706-965be02d81d0"). InnerVolumeSpecName "kube-api-access-v7ckk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:18.044220 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.044133 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-kube-api-access-ts9bv" pod "ded09c48-50f3-417f-9502-39403c2878d4" (UID: "ded09c48-50f3-417f-9502-39403c2878d4"). InnerVolumeSpecName "kube-api-access-ts9bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:18.044220 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.044142 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ded09c48-50f3-417f-9502-39403c2878d4-istio-podinfo" pod "ded09c48-50f3-417f-9502-39403c2878d4" (UID: "ded09c48-50f3-417f-9502-39403c2878d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:18.044220 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.044161 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-envoy" pod "ca5d24c0-427d-4b1c-b706-965be02d81d0" (UID: "ca5d24c0-427d-4b1c-b706-965be02d81d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:18.044325 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.044236 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-istio-token" pod "ded09c48-50f3-417f-9502-39403c2878d4" (UID: "ded09c48-50f3-417f-9502-39403c2878d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:18.079307 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.079271 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:18.099784 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099749 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.099784 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099779 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099794 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded09c48-50f3-417f-9502-39403c2878d4-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099808 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d182005-2390-4c46-8327-c051ce27933a-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099823 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts9bv\" (UniqueName: \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-kube-api-access-ts9bv\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099835 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099847 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099861 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099874 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099886 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded09c48-50f3-417f-9502-39403c2878d4-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099899 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099911 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099925 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded09c48-50f3-417f-9502-39403c2878d4-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099936 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099948 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f164162-c2b6-41de-9068-ca6eedf8417f-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099961 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4svw9\" (UniqueName: \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-kube-api-access-4svw9\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099974 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d182005-2390-4c46-8327-c051ce27933a-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.099988 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.100001 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.100014 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.100029 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d182005-2390-4c46-8327-c051ce27933a-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.100043 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100727 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.100058 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d182005-2390-4c46-8327-c051ce27933a-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100727 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.100071 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded09c48-50f3-417f-9502-39403c2878d4-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100727 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.100123 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100727 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.100137 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f164162-c2b6-41de-9068-ca6eedf8417f-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100727 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.100152 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5d24c0-427d-4b1c-b706-965be02d81d0-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100727 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.100166 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7ckk\" (UniqueName: \"kubernetes.io/projected/ca5d24c0-427d-4b1c-b706-965be02d81d0-kube-api-access-v7ckk\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.100727 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.100180 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76m8b\" (UniqueName: \"kubernetes.io/projected/3f164162-c2b6-41de-9068-ca6eedf8417f-kube-api-access-76m8b\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:18.255519 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.255492 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k"] Nov 02 18:00:18.258541 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.258517 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-t2s5k"] Nov 02 18:00:18.270714 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.270686 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb"] Nov 02 18:00:18.273629 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.273600 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-skqgb"] Nov 02 18:00:18.288123 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.288091 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj"] Nov 02 18:00:18.295275 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.295240 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-7lzcj"] Nov 02 18:00:18.308059 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.308034 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm"] Nov 02 18:00:18.313603 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.313579 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea752f-openshift-default-549fb46797-58xfm"] Nov 02 18:00:18.683360 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.683252 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d182005-2390-4c46-8327-c051ce27933a" path="/var/lib/kubelet/pods/2d182005-2390-4c46-8327-c051ce27933a/volumes" Nov 02 18:00:18.683957 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.683940 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" path="/var/lib/kubelet/pods/3f164162-c2b6-41de-9068-ca6eedf8417f/volumes" Nov 02 18:00:18.684534 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.684499 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" path="/var/lib/kubelet/pods/ca5d24c0-427d-4b1c-b706-965be02d81d0/volumes" Nov 02 18:00:18.685540 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.685492 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded09c48-50f3-417f-9502-39403c2878d4" path="/var/lib/kubelet/pods/ded09c48-50f3-417f-9502-39403c2878d4/volumes" Nov 02 18:00:18.744070 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.744035 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:18.848115 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.847790 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:18.946512 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.946420 4310 generic.go:356] "Generic (PLEG): container finished" podID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerID="ef8e22940caa76281ed7ea156c4c4d218f440e6825ff3cbe4bcbd99358d2e7e4" exitCode=60 Nov 02 18:00:18.946512 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.946496 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" event={"ID":"52879a72-ff92-4f16-83bd-0f65dd98019d","Type":"ContainerDied","Data":"ef8e22940caa76281ed7ea156c4c4d218f440e6825ff3cbe4bcbd99358d2e7e4"} Nov 02 18:00:18.946760 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.946600 4310 scope.go:122] "RemoveContainer" containerID="bb9810270ff02e5e24e6f70dfea86c612eb6b479d787595a94fd2ed92fac7223" Nov 02 18:00:18.947119 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.947069 4310 scope.go:122] "RemoveContainer" containerID="ef8e22940caa76281ed7ea156c4c4d218f440e6825ff3cbe4bcbd99358d2e7e4" Nov 02 18:00:18.947463 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:18.947437 4310 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4_odh-ai-gateway-infra(52879a72-ff92-4f16-83bd-0f65dd98019d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" Nov 02 18:00:18.995400 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:18.995369 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:19.028865 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.028832 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6"] Nov 02 18:00:19.216538 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.216511 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" Nov 02 18:00:19.331400 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.331361 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4bed6e2-a896-49b5-ad01-4b07d3861161-kube-api-access-fpq8h\" (UniqueName: \"kubernetes.io/projected/d4bed6e2-a896-49b5-ad01-4b07d3861161-kube-api-access-fpq8h\") pod \"d4bed6e2-a896-49b5-ad01-4b07d3861161\" (UID: \"d4bed6e2-a896-49b5-ad01-4b07d3861161\") " Nov 02 18:00:19.331861 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.331568 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4bed6e2-a896-49b5-ad01-4b07d3861161-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bed6e2-a896-49b5-ad01-4b07d3861161-maas-api-service-ca\") pod \"d4bed6e2-a896-49b5-ad01-4b07d3861161\" (UID: \"d4bed6e2-a896-49b5-ad01-4b07d3861161\") " Nov 02 18:00:19.331941 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.331882 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4bed6e2-a896-49b5-ad01-4b07d3861161-maas-api-service-ca" pod "d4bed6e2-a896-49b5-ad01-4b07d3861161" (UID: "d4bed6e2-a896-49b5-ad01-4b07d3861161"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:19.331992 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.331964 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bed6e2-a896-49b5-ad01-4b07d3861161-maas-api-service-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:19.333721 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.333696 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4bed6e2-a896-49b5-ad01-4b07d3861161-kube-api-access-fpq8h" pod "d4bed6e2-a896-49b5-ad01-4b07d3861161" (UID: "d4bed6e2-a896-49b5-ad01-4b07d3861161"). InnerVolumeSpecName "kube-api-access-fpq8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:19.433157 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.433118 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpq8h\" (UniqueName: \"kubernetes.io/projected/d4bed6e2-a896-49b5-ad01-4b07d3861161-kube-api-access-fpq8h\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:19.705012 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.704969 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:19.960688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.960603 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" event={"ID":"d4bed6e2-a896-49b5-ad01-4b07d3861161","Type":"ContainerDied","Data":"a400f904a47bccb1cb0e64be7dc2ccbbebe79057d8a5141b7648dea66da3a2ad"} Nov 02 18:00:19.960688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.960643 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6" Nov 02 18:00:19.960688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.960668 4310 scope.go:122] "RemoveContainer" containerID="2e0f362e1e653126f7732724be4001247570749c0b6d5aebb46892f3da854129" Nov 02 18:00:19.988104 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:19.987143 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4"] Nov 02 18:00:20.046325 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.042910 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6"] Nov 02 18:00:20.051252 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.049843 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-263a5ecf-29894040-xlmf6"] Nov 02 18:00:20.189692 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.189665 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" Nov 02 18:00:20.243654 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.243613 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52879a72-ff92-4f16-83bd-0f65dd98019d-kube-api-access-jv9g5\" (UniqueName: \"kubernetes.io/projected/52879a72-ff92-4f16-83bd-0f65dd98019d-kube-api-access-jv9g5\") pod \"52879a72-ff92-4f16-83bd-0f65dd98019d\" (UID: \"52879a72-ff92-4f16-83bd-0f65dd98019d\") " Nov 02 18:00:20.243890 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.243701 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52879a72-ff92-4f16-83bd-0f65dd98019d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52879a72-ff92-4f16-83bd-0f65dd98019d-maas-api-service-ca\") pod \"52879a72-ff92-4f16-83bd-0f65dd98019d\" (UID: \"52879a72-ff92-4f16-83bd-0f65dd98019d\") " Nov 02 18:00:20.244161 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.244132 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52879a72-ff92-4f16-83bd-0f65dd98019d-maas-api-service-ca" pod "52879a72-ff92-4f16-83bd-0f65dd98019d" (UID: "52879a72-ff92-4f16-83bd-0f65dd98019d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:20.244318 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.244300 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52879a72-ff92-4f16-83bd-0f65dd98019d-maas-api-service-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:20.245845 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.245820 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52879a72-ff92-4f16-83bd-0f65dd98019d-kube-api-access-jv9g5" pod "52879a72-ff92-4f16-83bd-0f65dd98019d" (UID: "52879a72-ff92-4f16-83bd-0f65dd98019d"). InnerVolumeSpecName "kube-api-access-jv9g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:20.345810 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.345767 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv9g5\" (UniqueName: \"kubernetes.io/projected/52879a72-ff92-4f16-83bd-0f65dd98019d-kube-api-access-jv9g5\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:20.485520 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.485483 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:20.674826 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.674744 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" path="/var/lib/kubelet/pods/d4bed6e2-a896-49b5-ad01-4b07d3861161/volumes" Nov 02 18:00:20.947863 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.947773 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:20.968805 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.968777 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" Nov 02 18:00:20.968988 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.968793 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4" event={"ID":"52879a72-ff92-4f16-83bd-0f65dd98019d","Type":"ContainerDied","Data":"aa15a534ee1cdcf9eaf086d855e6089b3994af7605c3319efa145d5236ff3360"} Nov 02 18:00:20.968988 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.968928 4310 scope.go:122] "RemoveContainer" containerID="ef8e22940caa76281ed7ea156c4c4d218f440e6825ff3cbe4bcbd99358d2e7e4" Nov 02 18:00:20.988640 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.988615 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4"] Nov 02 18:00:20.994457 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:20.994426 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb4ecb-29894040-ffgt4"] Nov 02 18:00:21.185436 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:21.185403 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:22.088981 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:22.088947 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:22.675248 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:22.675217 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" path="/var/lib/kubelet/pods/52879a72-ff92-4f16-83bd-0f65dd98019d/volumes" Nov 02 18:00:22.894244 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:22.894202 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:23.594832 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:23.594798 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:24.281139 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:24.281104 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:24.412596 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:24.412562 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:24.983669 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:24.983632 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:25.696019 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:25.695991 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:26.459103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:26.455292 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:27.290450 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:27.290416 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:27.441542 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:27.441503 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:27.887452 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:27.887414 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:28.032231 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:28.032205 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7"] Nov 02 18:00:28.034792 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:28.034768 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21dc3425-bmgq7"] Nov 02 18:00:28.668720 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:28.668662 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:28.674298 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:28.674270 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee5218f-23ea-478e-925b-f324037224dc" path="/var/lib/kubelet/pods/cee5218f-23ea-478e-925b-f324037224dc/volumes" Nov 02 18:00:29.376696 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:29.376660 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:30.170479 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:30.170443 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:30.875415 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:30.875382 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:31.582813 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:31.582775 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:32.368889 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:32.368854 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:33.044933 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:33.044895 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:33.522962 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:33.522921 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:00:33.684746 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:33.684712 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:34.386879 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.386843 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:34.739787 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.739751 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d"] Nov 02 18:00:34.740277 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740249 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerName="cleanup" Nov 02 18:00:34.740277 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740271 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerName="cleanup" Nov 02 18:00:34.740277 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740284 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerName="cleanup" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740289 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerName="cleanup" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740297 4310 cpu_manager.go:414] "containerMap: removing container" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerName="cleanup" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740302 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerName="cleanup" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740320 4310 cpu_manager.go:414] "containerMap: removing container" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740325 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740334 4310 cpu_manager.go:414] "containerMap: removing container" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740338 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740348 4310 cpu_manager.go:414] "containerMap: removing container" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerName="cleanup" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740352 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerName="cleanup" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740364 4310 cpu_manager.go:414] "containerMap: removing container" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740372 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740380 4310 cpu_manager.go:414] "containerMap: removing container" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740385 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740395 4310 cpu_manager.go:414] "containerMap: removing container" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740400 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740410 4310 cpu_manager.go:414] "containerMap: removing container" podUID="12043acc-60b0-48d1-9164-127fc5215f28" containerName="payload-processing" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740414 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12043acc-60b0-48d1-9164-127fc5215f28" containerName="payload-processing" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740481 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerName="cleanup" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740487 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerName="cleanup" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740493 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerName="cleanup" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740505 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded09c48-50f3-417f-9502-39403c2878d4" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740513 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerName="cleanup" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740520 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740529 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d182005-2390-4c46-8327-c051ce27933a" containerName="istio-proxy" Nov 02 18:00:34.740528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740537 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f164162-c2b6-41de-9068-ca6eedf8417f" containerName="istio-proxy" Nov 02 18:00:34.741478 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740545 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5d24c0-427d-4b1c-b706-965be02d81d0" containerName="istio-proxy" Nov 02 18:00:34.741478 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.740552 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="12043acc-60b0-48d1-9164-127fc5215f28" containerName="payload-processing" Nov 02 18:00:34.744093 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.744060 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:34.750905 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.750877 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:00:34.754736 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.754712 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d"] Nov 02 18:00:34.853341 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.853309 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:34.853473 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.853363 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq2v4\" (UniqueName: \"kubernetes.io/projected/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-kube-api-access-bq2v4\") pod \"maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:34.853473 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.853421 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:34.957762 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.954749 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:34.957762 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.954831 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq2v4\" (UniqueName: \"kubernetes.io/projected/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-kube-api-access-bq2v4\") pod \"maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:34.957762 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.954860 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:34.957762 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.955629 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:34.957762 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.957313 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:34.964472 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:34.964437 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq2v4\" (UniqueName: \"kubernetes.io/projected/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-kube-api-access-bq2v4\") pod \"maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:35.017846 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.017817 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" Nov 02 18:00:35.055886 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.055847 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c2061c-374e-4200-b30c-694f64cfb127-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c2061c-374e-4200-b30c-694f64cfb127-maas-api-service-ca\") pod \"a4c2061c-374e-4200-b30c-694f64cfb127\" (UID: \"a4c2061c-374e-4200-b30c-694f64cfb127\") " Nov 02 18:00:35.056071 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.055913 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c2061c-374e-4200-b30c-694f64cfb127-kube-api-access-mfllp\" (UniqueName: \"kubernetes.io/projected/a4c2061c-374e-4200-b30c-694f64cfb127-kube-api-access-mfllp\") pod \"a4c2061c-374e-4200-b30c-694f64cfb127\" (UID: \"a4c2061c-374e-4200-b30c-694f64cfb127\") " Nov 02 18:00:35.056344 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.056310 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c2061c-374e-4200-b30c-694f64cfb127-maas-api-service-ca" pod "a4c2061c-374e-4200-b30c-694f64cfb127" (UID: "a4c2061c-374e-4200-b30c-694f64cfb127"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:35.056551 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.056529 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c2061c-374e-4200-b30c-694f64cfb127-maas-api-service-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:35.059254 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.059105 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c2061c-374e-4200-b30c-694f64cfb127-kube-api-access-mfllp" pod "a4c2061c-374e-4200-b30c-694f64cfb127" (UID: "a4c2061c-374e-4200-b30c-694f64cfb127"). InnerVolumeSpecName "kube-api-access-mfllp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:35.062232 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.062211 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:35.083162 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.083120 4310 generic.go:356] "Generic (PLEG): container finished" podID="a4c2061c-374e-4200-b30c-694f64cfb127" containerID="6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4" exitCode=137 Nov 02 18:00:35.083298 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.083199 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" Nov 02 18:00:35.083298 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.083202 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" event={"ID":"a4c2061c-374e-4200-b30c-694f64cfb127","Type":"ContainerDied","Data":"6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4"} Nov 02 18:00:35.083298 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.083259 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8" event={"ID":"a4c2061c-374e-4200-b30c-694f64cfb127","Type":"ContainerDied","Data":"309f5c4ad4d9bff2aa7056f6fb9f47283dd22b2222d6dbdaba74fae9d15162ae"} Nov 02 18:00:35.083298 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.083283 4310 scope.go:122] "RemoveContainer" containerID="6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4" Nov 02 18:00:35.095759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.095734 4310 scope.go:122] "RemoveContainer" containerID="91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365" Nov 02 18:00:35.107007 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.106986 4310 scope.go:122] "RemoveContainer" containerID="6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4" Nov 02 18:00:35.107432 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:35.107411 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4\": container with ID starting with 6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4 not found: ID does not exist" containerID="6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4" Nov 02 18:00:35.107510 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.107445 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4"} err="failed to get container status \"6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4\": rpc error: code = NotFound desc = could not find container \"6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4\": container with ID starting with 6807735317d16d608b8b6de5ee41a90c670ac2341ce8adbd77b4c04a97a330f4 not found: ID does not exist" Nov 02 18:00:35.107510 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.107463 4310 scope.go:122] "RemoveContainer" containerID="91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365" Nov 02 18:00:35.107849 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:35.107815 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365\": container with ID starting with 91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365 not found: ID does not exist" containerID="91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365" Nov 02 18:00:35.107890 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.107863 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365"} err="failed to get container status \"91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365\": rpc error: code = NotFound desc = could not find container \"91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365\": container with ID starting with 91a53d3644cf6bf3c4719f590c8d596b2adb9edf491d9d3a6e56e899a0165365 not found: ID does not exist" Nov 02 18:00:35.110704 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.110668 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8"] Nov 02 18:00:35.114034 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.114007 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea752f-29894040-sj7g8"] Nov 02 18:00:35.157553 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.157514 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfllp\" (UniqueName: \"kubernetes.io/projected/a4c2061c-374e-4200-b30c-694f64cfb127-kube-api-access-mfllp\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:35.267172 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.267072 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:35.375107 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:00:35.375060 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902ae94e_7b7f_42ba_a9e3_d8c2db9ba71f.slice/crio-3070739ec0af5fee6d56f0d59ed7bf8e48c8ac06fbdafa9a02cde688f3fb4aed WatchSource:0}: Error finding container 3070739ec0af5fee6d56f0d59ed7bf8e48c8ac06fbdafa9a02cde688f3fb4aed: Status 404 returned error can't find the container with id 3070739ec0af5fee6d56f0d59ed7bf8e48c8ac06fbdafa9a02cde688f3fb4aed Nov 02 18:00:35.375409 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.375386 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d"] Nov 02 18:00:35.981774 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:35.981738 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:36.091314 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:36.091213 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" event={"ID":"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f","Type":"ContainerStarted","Data":"8380e2ae66283467884a17c6319fdfa2b24bee5b3ea8e3b5f37c85feeb98b947"} Nov 02 18:00:36.091314 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:36.091271 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" event={"ID":"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f","Type":"ContainerStarted","Data":"3070739ec0af5fee6d56f0d59ed7bf8e48c8ac06fbdafa9a02cde688f3fb4aed"} Nov 02 18:00:36.124105 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:36.111996 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" podStartSLOduration=2.111980201 podStartE2EDuration="2.111980201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:36.109327514 +0000 UTC m=+1233.552410150" watchObservedRunningTime="2026-11-02 18:00:36.111980201 +0000 UTC m=+1233.555062837" Nov 02 18:00:36.687254 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:36.687218 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c2061c-374e-4200-b30c-694f64cfb127" path="/var/lib/kubelet/pods/a4c2061c-374e-4200-b30c-694f64cfb127/volumes" Nov 02 18:00:36.792294 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:36.792257 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:37.101205 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:37.101167 4310 generic.go:356] "Generic (PLEG): container finished" podID="902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f" containerID="8380e2ae66283467884a17c6319fdfa2b24bee5b3ea8e3b5f37c85feeb98b947" exitCode=0 Nov 02 18:00:37.101660 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:37.101253 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" event={"ID":"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f","Type":"ContainerDied","Data":"8380e2ae66283467884a17c6319fdfa2b24bee5b3ea8e3b5f37c85feeb98b947"} Nov 02 18:00:37.676220 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:37.676187 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:38.257681 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.257645 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn"] Nov 02 18:00:38.258309 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.257935 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" containerID="cri-o://484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731" gracePeriod=30 Nov 02 18:00:38.261852 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.260907 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2"] Nov 02 18:00:38.261852 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.261663 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" containerID="cri-o://adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea" gracePeriod=30 Nov 02 18:00:38.266220 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.266198 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs"] Nov 02 18:00:38.266442 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.266419 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" containerID="cri-o://036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f" gracePeriod=30 Nov 02 18:00:38.273460 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.273436 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:38.291289 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.291262 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-kube-api-access-bq2v4\" (UniqueName: \"kubernetes.io/projected/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-kube-api-access-bq2v4\") pod \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " Nov 02 18:00:38.291482 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.291311 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-openshift-ca\") pod \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " Nov 02 18:00:38.291482 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.291428 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-xks-ca\") pod \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\" (UID: \"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f\") " Nov 02 18:00:38.291879 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.291854 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-openshift-ca" pod "902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f" (UID: "902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:38.293577 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.293545 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-kube-api-access-bq2v4" pod "902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f" (UID: "902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f"). InnerVolumeSpecName "kube-api-access-bq2v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:38.293970 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.293944 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-xks-ca" pod "902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f" (UID: "902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:00:38.393474 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.393373 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:38.393474 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.393406 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq2v4\" (UniqueName: \"kubernetes.io/projected/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-kube-api-access-bq2v4\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:38.393474 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.393421 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:38.399590 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:38.399558 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:39.076695 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:39.076654 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:39.137931 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:39.137890 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" event={"ID":"902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f","Type":"ContainerDied","Data":"3070739ec0af5fee6d56f0d59ed7bf8e48c8ac06fbdafa9a02cde688f3fb4aed"} Nov 02 18:00:39.138164 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:39.137979 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3070739ec0af5fee6d56f0d59ed7bf8e48c8ac06fbdafa9a02cde688f3fb4aed" Nov 02 18:00:39.138164 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:39.138048 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d" Nov 02 18:00:39.883714 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:39.883656 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:40.590660 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:40.590614 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:41.272759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:41.272720 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:42.026249 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:42.026214 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6"] Nov 02 18:00:42.032531 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:42.032495 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n"] Nov 02 18:00:42.035370 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:42.035342 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-719e5d3e-7bkd6"] Nov 02 18:00:42.037630 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:42.037608 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdcf90d-ghl5n"] Nov 02 18:00:42.082868 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:42.082834 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:42.673974 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:42.673944 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69353ed1-5667-4407-a4bc-36105ebcb410" path="/var/lib/kubelet/pods/69353ed1-5667-4407-a4bc-36105ebcb410/volumes" Nov 02 18:00:42.674411 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:42.674396 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ef6ff4-a23e-45b0-900e-3c3c057df7cd" path="/var/lib/kubelet/pods/c3ef6ff4-a23e-45b0-900e-3c3c057df7cd/volumes" Nov 02 18:00:42.780804 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:42.780770 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:43.540696 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.540623 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 18:00:43.575861 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.575832 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 18:00:43.581767 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.581735 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 18:00:43.595525 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.595498 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:43.659269 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659235 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-podinfo\") pod \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " Nov 02 18:00:43.659443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659282 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-podinfo\") pod \"10d03bf3-ecba-4349-ac6c-0410e573d494\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " Nov 02 18:00:43.659443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659307 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13401d6a-300b-4bee-beef-fd0fb068e112-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13401d6a-300b-4bee-beef-fd0fb068e112-istio-podinfo\") pod \"13401d6a-300b-4bee-beef-fd0fb068e112\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " Nov 02 18:00:43.659443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659335 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-socket\") pod \"13401d6a-300b-4bee-beef-fd0fb068e112\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " Nov 02 18:00:43.659443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659376 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-envoy\") pod \"13401d6a-300b-4bee-beef-fd0fb068e112\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " Nov 02 18:00:43.659443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659401 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-token\" (UniqueName: \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-token\") pod \"10d03bf3-ecba-4349-ac6c-0410e573d494\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " Nov 02 18:00:43.659443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659425 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-socket\") pod \"10d03bf3-ecba-4349-ac6c-0410e573d494\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " Nov 02 18:00:43.659764 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659711 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-socket" pod "13401d6a-300b-4bee-beef-fd0fb068e112" (UID: "13401d6a-300b-4bee-beef-fd0fb068e112"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.659764 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659716 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-socket" pod "10d03bf3-ecba-4349-ac6c-0410e573d494" (UID: "10d03bf3-ecba-4349-ac6c-0410e573d494"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.659862 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659770 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istiod-ca-cert\") pod \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " Nov 02 18:00:43.659862 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659823 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-kube-api-access-79gn2\" (UniqueName: \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-kube-api-access-79gn2\") pod \"10d03bf3-ecba-4349-ac6c-0410e573d494\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " Nov 02 18:00:43.659967 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659860 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-envoy\") pod \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " Nov 02 18:00:43.659967 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659893 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13401d6a-300b-4bee-beef-fd0fb068e112-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13401d6a-300b-4bee-beef-fd0fb068e112-istiod-ca-cert\") pod \"13401d6a-300b-4bee-beef-fd0fb068e112\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " Nov 02 18:00:43.659967 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659922 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-envoy\") pod \"10d03bf3-ecba-4349-ac6c-0410e573d494\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " Nov 02 18:00:43.660188 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.659950 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-data\") pod \"13401d6a-300b-4bee-beef-fd0fb068e112\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " Nov 02 18:00:43.660188 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660016 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-credential-socket\") pod \"10d03bf3-ecba-4349-ac6c-0410e573d494\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " Nov 02 18:00:43.660188 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660045 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-certs\") pod \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " Nov 02 18:00:43.660188 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660123 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-socket\") pod \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " Nov 02 18:00:43.660188 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660162 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-data\") pod \"10d03bf3-ecba-4349-ac6c-0410e573d494\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " Nov 02 18:00:43.660188 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660163 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istiod-ca-cert" pod "0558ce75-6fbc-442b-8024-6b66f00e5f9f" (UID: "0558ce75-6fbc-442b-8024-6b66f00e5f9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:43.660511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660193 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-certs\") pod \"13401d6a-300b-4bee-beef-fd0fb068e112\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " Nov 02 18:00:43.660511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660224 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-certs\") pod \"10d03bf3-ecba-4349-ac6c-0410e573d494\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " Nov 02 18:00:43.660511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660254 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-token\" (UniqueName: \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-token\") pod \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " Nov 02 18:00:43.660511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660343 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d03bf3-ecba-4349-ac6c-0410e573d494-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10d03bf3-ecba-4349-ac6c-0410e573d494-istiod-ca-cert\") pod \"10d03bf3-ecba-4349-ac6c-0410e573d494\" (UID: \"10d03bf3-ecba-4349-ac6c-0410e573d494\") " Nov 02 18:00:43.660511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660369 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-data\") pod \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " Nov 02 18:00:43.660511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660408 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-istio-token\" (UniqueName: \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-istio-token\") pod \"13401d6a-300b-4bee-beef-fd0fb068e112\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " Nov 02 18:00:43.660511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660420 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-data" pod "13401d6a-300b-4bee-beef-fd0fb068e112" (UID: "13401d6a-300b-4bee-beef-fd0fb068e112"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.660511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660475 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-kube-api-access-l4rtz\" (UniqueName: \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-kube-api-access-l4rtz\") pod \"13401d6a-300b-4bee-beef-fd0fb068e112\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " Nov 02 18:00:43.660918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660526 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-credential-socket\") pod \"13401d6a-300b-4bee-beef-fd0fb068e112\" (UID: \"13401d6a-300b-4bee-beef-fd0fb068e112\") " Nov 02 18:00:43.660918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660581 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-credential-socket\") pod \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " Nov 02 18:00:43.660918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.660620 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-kube-api-access-vv5k2\" (UniqueName: \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-kube-api-access-vv5k2\") pod \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\" (UID: \"0558ce75-6fbc-442b-8024-6b66f00e5f9f\") " Nov 02 18:00:43.661152 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.661128 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.661219 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.661156 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.661219 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.661171 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.661219 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.661185 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.661987 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.661964 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13401d6a-300b-4bee-beef-fd0fb068e112-istio-podinfo" pod "13401d6a-300b-4bee-beef-fd0fb068e112" (UID: "13401d6a-300b-4bee-beef-fd0fb068e112"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:43.662355 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.662239 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-podinfo" pod "10d03bf3-ecba-4349-ac6c-0410e573d494" (UID: "10d03bf3-ecba-4349-ac6c-0410e573d494"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:43.662916 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.662889 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-envoy" pod "13401d6a-300b-4bee-beef-fd0fb068e112" (UID: "13401d6a-300b-4bee-beef-fd0fb068e112"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.663654 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.663626 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-kube-api-access-vv5k2" pod "0558ce75-6fbc-442b-8024-6b66f00e5f9f" (UID: "0558ce75-6fbc-442b-8024-6b66f00e5f9f"). InnerVolumeSpecName "kube-api-access-vv5k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:43.663988 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.663964 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d03bf3-ecba-4349-ac6c-0410e573d494-istiod-ca-cert" pod "10d03bf3-ecba-4349-ac6c-0410e573d494" (UID: "10d03bf3-ecba-4349-ac6c-0410e573d494"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:43.664273 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.664249 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-data" pod "0558ce75-6fbc-442b-8024-6b66f00e5f9f" (UID: "0558ce75-6fbc-442b-8024-6b66f00e5f9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.664911 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.664889 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-envoy" pod "0558ce75-6fbc-442b-8024-6b66f00e5f9f" (UID: "0558ce75-6fbc-442b-8024-6b66f00e5f9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.665033 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.664908 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-envoy" pod "10d03bf3-ecba-4349-ac6c-0410e573d494" (UID: "10d03bf3-ecba-4349-ac6c-0410e573d494"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.665173 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.665070 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-token" pod "0558ce75-6fbc-442b-8024-6b66f00e5f9f" (UID: "0558ce75-6fbc-442b-8024-6b66f00e5f9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:43.665264 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.665175 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-socket" pod "0558ce75-6fbc-442b-8024-6b66f00e5f9f" (UID: "0558ce75-6fbc-442b-8024-6b66f00e5f9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.665264 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.665183 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-credential-socket" pod "10d03bf3-ecba-4349-ac6c-0410e573d494" (UID: "10d03bf3-ecba-4349-ac6c-0410e573d494"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.665427 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.665287 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-data" pod "10d03bf3-ecba-4349-ac6c-0410e573d494" (UID: "10d03bf3-ecba-4349-ac6c-0410e573d494"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.665427 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.665413 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-certs" pod "13401d6a-300b-4bee-beef-fd0fb068e112" (UID: "13401d6a-300b-4bee-beef-fd0fb068e112"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.665586 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.665422 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-certs" pod "0558ce75-6fbc-442b-8024-6b66f00e5f9f" (UID: "0558ce75-6fbc-442b-8024-6b66f00e5f9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.665586 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.665484 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-certs" pod "10d03bf3-ecba-4349-ac6c-0410e573d494" (UID: "10d03bf3-ecba-4349-ac6c-0410e573d494"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.665733 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.665646 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-credential-socket" pod "0558ce75-6fbc-442b-8024-6b66f00e5f9f" (UID: "0558ce75-6fbc-442b-8024-6b66f00e5f9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.665733 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.665651 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-credential-socket" pod "13401d6a-300b-4bee-beef-fd0fb068e112" (UID: "13401d6a-300b-4bee-beef-fd0fb068e112"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:00:43.665890 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.665779 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-token" pod "10d03bf3-ecba-4349-ac6c-0410e573d494" (UID: "10d03bf3-ecba-4349-ac6c-0410e573d494"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:43.667590 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.667566 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-kube-api-access-l4rtz" pod "13401d6a-300b-4bee-beef-fd0fb068e112" (UID: "13401d6a-300b-4bee-beef-fd0fb068e112"). InnerVolumeSpecName "kube-api-access-l4rtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:43.668970 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.668945 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-istio-token" pod "13401d6a-300b-4bee-beef-fd0fb068e112" (UID: "13401d6a-300b-4bee-beef-fd0fb068e112"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:43.671489 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.671464 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-kube-api-access-79gn2" pod "10d03bf3-ecba-4349-ac6c-0410e573d494" (UID: "10d03bf3-ecba-4349-ac6c-0410e573d494"). InnerVolumeSpecName "kube-api-access-79gn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:43.672645 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.672620 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-podinfo" pod "0558ce75-6fbc-442b-8024-6b66f00e5f9f" (UID: "0558ce75-6fbc-442b-8024-6b66f00e5f9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:00:43.673291 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.673263 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13401d6a-300b-4bee-beef-fd0fb068e112-istiod-ca-cert" pod "13401d6a-300b-4bee-beef-fd0fb068e112" (UID: "13401d6a-300b-4bee-beef-fd0fb068e112"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763213 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763250 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4rtz\" (UniqueName: \"kubernetes.io/projected/13401d6a-300b-4bee-beef-fd0fb068e112-kube-api-access-l4rtz\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763264 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763277 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763290 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv5k2\" (UniqueName: \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-kube-api-access-vv5k2\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763302 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763316 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763328 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13401d6a-300b-4bee-beef-fd0fb068e112-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763340 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763352 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763365 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79gn2\" (UniqueName: \"kubernetes.io/projected/10d03bf3-ecba-4349-ac6c-0410e573d494-kube-api-access-79gn2\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763380 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763395 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13401d6a-300b-4bee-beef-fd0fb068e112-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763408 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763425 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763438 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763451 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763464 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763475 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13401d6a-300b-4bee-beef-fd0fb068e112-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763489 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10d03bf3-ecba-4349-ac6c-0410e573d494-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763502 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763516 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10d03bf3-ecba-4349-ac6c-0410e573d494-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:43.764730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:43.763529 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0558ce75-6fbc-442b-8024-6b66f00e5f9f-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:44.178776 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.178745 4310 generic.go:356] "Generic (PLEG): container finished" podID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerID="036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f" exitCode=0 Nov 02 18:00:44.178975 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.178811 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" Nov 02 18:00:44.178975 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.178831 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" event={"ID":"0558ce75-6fbc-442b-8024-6b66f00e5f9f","Type":"ContainerDied","Data":"036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f"} Nov 02 18:00:44.178975 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.178910 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs" event={"ID":"0558ce75-6fbc-442b-8024-6b66f00e5f9f","Type":"ContainerDied","Data":"7aa6a9651464a89bc190e4aa732a0ee840f906c00ffefadfc39a1a1dc3e0860d"} Nov 02 18:00:44.178975 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.178944 4310 scope.go:122] "RemoveContainer" containerID="036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f" Nov 02 18:00:44.180606 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.180579 4310 generic.go:356] "Generic (PLEG): container finished" podID="13401d6a-300b-4bee-beef-fd0fb068e112" containerID="484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731" exitCode=0 Nov 02 18:00:44.180737 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.180639 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" Nov 02 18:00:44.180737 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.180659 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" event={"ID":"13401d6a-300b-4bee-beef-fd0fb068e112","Type":"ContainerDied","Data":"484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731"} Nov 02 18:00:44.180737 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.180707 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn" event={"ID":"13401d6a-300b-4bee-beef-fd0fb068e112","Type":"ContainerDied","Data":"66761905a22d48de32eff26d72a866c0aa50067f2b692073679c78a2cce08aee"} Nov 02 18:00:44.182480 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.182381 4310 generic.go:356] "Generic (PLEG): container finished" podID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerID="adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea" exitCode=0 Nov 02 18:00:44.182480 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.182410 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" event={"ID":"10d03bf3-ecba-4349-ac6c-0410e573d494","Type":"ContainerDied","Data":"adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea"} Nov 02 18:00:44.182480 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.182427 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" Nov 02 18:00:44.182480 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.182457 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2" event={"ID":"10d03bf3-ecba-4349-ac6c-0410e573d494","Type":"ContainerDied","Data":"0195302e3687a114d04c00c0b91f4946fd43986c0c760dccd4ca2ce84b974ba5"} Nov 02 18:00:44.194017 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.193986 4310 scope.go:122] "RemoveContainer" containerID="036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f" Nov 02 18:00:44.194606 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:44.194577 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f\": container with ID starting with 036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f not found: ID does not exist" containerID="036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f" Nov 02 18:00:44.194720 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.194615 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f"} err="failed to get container status \"036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f\": rpc error: code = NotFound desc = could not find container \"036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f\": container with ID starting with 036048c322d96abaf880822bc48371328ed636e8b82588bb1e1e6819d52b557f not found: ID does not exist" Nov 02 18:00:44.194720 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.194640 4310 scope.go:122] "RemoveContainer" containerID="484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731" Nov 02 18:00:44.205290 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.205264 4310 scope.go:122] "RemoveContainer" containerID="484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731" Nov 02 18:00:44.205641 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:44.205618 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731\": container with ID starting with 484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731 not found: ID does not exist" containerID="484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731" Nov 02 18:00:44.205735 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.205654 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731"} err="failed to get container status \"484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731\": rpc error: code = NotFound desc = could not find container \"484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731\": container with ID starting with 484616dc6b7fff6c3b730b2f6e053980c11770ad44e25726f05db16c1b430731 not found: ID does not exist" Nov 02 18:00:44.205735 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.205679 4310 scope.go:122] "RemoveContainer" containerID="adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea" Nov 02 18:00:44.213686 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.213660 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2"] Nov 02 18:00:44.216527 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.216504 4310 scope.go:122] "RemoveContainer" containerID="adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea" Nov 02 18:00:44.216845 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:44.216826 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea\": container with ID starting with adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea not found: ID does not exist" containerID="adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea" Nov 02 18:00:44.216922 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.216852 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea"} err="failed to get container status \"adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea\": rpc error: code = NotFound desc = could not find container \"adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea\": container with ID starting with adfa5b82c995517444993c7919beba8a0e3319783ec84e21dcae8c1ccc31f5ea not found: ID does not exist" Nov 02 18:00:44.217487 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.217467 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-w2dl2"] Nov 02 18:00:44.232894 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.232863 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn"] Nov 02 18:00:44.238626 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.238602 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-s5kgn"] Nov 02 18:00:44.251513 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.251488 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs"] Nov 02 18:00:44.256392 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.256366 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-knxfs"] Nov 02 18:00:44.290586 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.290556 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:44.672639 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.672604 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" path="/var/lib/kubelet/pods/0558ce75-6fbc-442b-8024-6b66f00e5f9f/volumes" Nov 02 18:00:44.673061 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.673048 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" path="/var/lib/kubelet/pods/10d03bf3-ecba-4349-ac6c-0410e573d494/volumes" Nov 02 18:00:44.673483 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.673456 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" path="/var/lib/kubelet/pods/13401d6a-300b-4bee-beef-fd0fb068e112/volumes" Nov 02 18:00:44.901108 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.901033 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:44.915202 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.915165 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z"] Nov 02 18:00:44.998691 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:44.998630 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:45.203397 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:45.203361 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:45.295027 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:45.294923 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d"] Nov 02 18:00:45.297981 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:45.297947 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-263a5ecf-zqj2d"] Nov 02 18:00:45.318687 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:45.318645 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:45.409898 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:45.409863 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:46.091060 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:46.091021 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:46.675505 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:46.675472 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f" path="/var/lib/kubelet/pods/902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f/volumes" Nov 02 18:00:46.802018 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:46.801984 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:47.562509 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:47.562471 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:48.042250 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:48.042199 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:48.489784 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:48.489748 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:49.386523 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:49.386496 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:49.916365 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:49.916299 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" podUID="55222bef-563e-447e-af4b-ea8cc8fa190f" containerName="payload-pre-processing" containerID="cri-o://8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17" gracePeriod=25 Nov 02 18:00:50.175212 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.175146 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:50.176639 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.176617 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 18:00:50.251324 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.251271 4310 generic.go:356] "Generic (PLEG): container finished" podID="55222bef-563e-447e-af4b-ea8cc8fa190f" containerID="8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17" exitCode=0 Nov 02 18:00:50.251524 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.251333 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" Nov 02 18:00:50.252554 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.251618 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" event={"ID":"55222bef-563e-447e-af4b-ea8cc8fa190f","Type":"ContainerDied","Data":"8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17"} Nov 02 18:00:50.252554 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.251715 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z" event={"ID":"55222bef-563e-447e-af4b-ea8cc8fa190f","Type":"ContainerDied","Data":"74f7479f9662a009f71838c2f4f23bac2563a58776660f5397e61c78cc74a876"} Nov 02 18:00:50.252554 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.251747 4310 scope.go:122] "RemoveContainer" containerID="8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17" Nov 02 18:00:50.252957 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.252933 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55222bef-563e-447e-af4b-ea8cc8fa190f-kube-api-access-55hbk\" (UniqueName: \"kubernetes.io/projected/55222bef-563e-447e-af4b-ea8cc8fa190f-kube-api-access-55hbk\") pod \"55222bef-563e-447e-af4b-ea8cc8fa190f\" (UID: \"55222bef-563e-447e-af4b-ea8cc8fa190f\") " Nov 02 18:00:50.253058 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.253044 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55222bef-563e-447e-af4b-ea8cc8fa190f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55222bef-563e-447e-af4b-ea8cc8fa190f-plugins-config-volume\") pod \"55222bef-563e-447e-af4b-ea8cc8fa190f\" (UID: \"55222bef-563e-447e-af4b-ea8cc8fa190f\") " Nov 02 18:00:50.253439 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.253416 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55222bef-563e-447e-af4b-ea8cc8fa190f-plugins-config-volume" pod "55222bef-563e-447e-af4b-ea8cc8fa190f" (UID: "55222bef-563e-447e-af4b-ea8cc8fa190f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:50.255123 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.255099 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55222bef-563e-447e-af4b-ea8cc8fa190f-kube-api-access-55hbk" pod "55222bef-563e-447e-af4b-ea8cc8fa190f" (UID: "55222bef-563e-447e-af4b-ea8cc8fa190f"). InnerVolumeSpecName "kube-api-access-55hbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:50.269582 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.269562 4310 scope.go:122] "RemoveContainer" containerID="8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17" Nov 02 18:00:50.269875 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:00:50.269856 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17\": container with ID starting with 8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17 not found: ID does not exist" containerID="8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17" Nov 02 18:00:50.269956 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.269886 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17"} err="failed to get container status \"8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17\": rpc error: code = NotFound desc = could not find container \"8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17\": container with ID starting with 8699d05ac11a07e44c781397a50d76f9ba2b11cb676d405b215a2ac8918bcf17 not found: ID does not exist" Nov 02 18:00:50.354743 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.354703 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55hbk\" (UniqueName: \"kubernetes.io/projected/55222bef-563e-447e-af4b-ea8cc8fa190f-kube-api-access-55hbk\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:50.354743 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.354735 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55222bef-563e-447e-af4b-ea8cc8fa190f-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:00:50.579542 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.579505 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z"] Nov 02 18:00:50.584781 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.584741 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-263a5ecf-d47dcf979-8fr9z"] Nov 02 18:00:50.674288 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.674251 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55222bef-563e-447e-af4b-ea8cc8fa190f" path="/var/lib/kubelet/pods/55222bef-563e-447e-af4b-ea8cc8fa190f/volumes" Nov 02 18:00:50.918761 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:50.918662 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:51.676464 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:51.676431 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:52.379779 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:52.379714 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:53.124118 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:53.123501 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:53.795393 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:53.795359 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:54.491996 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:54.491960 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:55.185383 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:55.185350 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:55.800938 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:55.800902 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:55.901420 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:55.901386 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:56.704190 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:56.704103 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:57.461286 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:57.461247 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:58.271492 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:58.271464 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:58.981826 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:58.981783 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:00:59.678808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:00:59.678762 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:00.479486 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:00.479450 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:01.321857 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:01.321821 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:01.972458 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:01.972422 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:02.684910 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:02.684876 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:03.110821 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:03.110785 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:03.212582 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:03.212546 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:03.487750 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:03.487718 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:03.521705 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:01:03.521670 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:01:03.841706 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:03.841622 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:03.954485 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:03.954451 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:04.107941 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.107855 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4"] Nov 02 18:01:04.108304 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108291 4310 cpu_manager.go:414] "containerMap: removing container" podUID="55222bef-563e-447e-af4b-ea8cc8fa190f" containerName="payload-pre-processing" Nov 02 18:01:04.108362 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108305 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55222bef-563e-447e-af4b-ea8cc8fa190f" containerName="payload-pre-processing" Nov 02 18:01:04.108362 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108319 4310 cpu_manager.go:414] "containerMap: removing container" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" Nov 02 18:01:04.108362 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108324 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" Nov 02 18:01:04.108362 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108334 4310 cpu_manager.go:414] "containerMap: removing container" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" Nov 02 18:01:04.108362 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108339 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" Nov 02 18:01:04.108362 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108348 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a4c2061c-374e-4200-b30c-694f64cfb127" containerName="cleanup" Nov 02 18:01:04.108362 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108352 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c2061c-374e-4200-b30c-694f64cfb127" containerName="cleanup" Nov 02 18:01:04.108362 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108359 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a4c2061c-374e-4200-b30c-694f64cfb127" containerName="cleanup" Nov 02 18:01:04.108362 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108364 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c2061c-374e-4200-b30c-694f64cfb127" containerName="cleanup" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108374 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerName="cleanup" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108381 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerName="cleanup" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108387 4310 cpu_manager.go:414] "containerMap: removing container" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerName="cleanup" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108391 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerName="cleanup" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108401 4310 cpu_manager.go:414] "containerMap: removing container" podUID="902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f" containerName="revoke-keys" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108405 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f" containerName="revoke-keys" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108418 4310 cpu_manager.go:414] "containerMap: removing container" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108422 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108479 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="55222bef-563e-447e-af4b-ea8cc8fa190f" containerName="payload-pre-processing" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108489 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d03bf3-ecba-4349-ac6c-0410e573d494" containerName="istio-proxy" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108494 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="13401d6a-300b-4bee-beef-fd0fb068e112" containerName="istio-proxy" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108501 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c2061c-374e-4200-b30c-694f64cfb127" containerName="cleanup" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108506 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c2061c-374e-4200-b30c-694f64cfb127" containerName="cleanup" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108512 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4bed6e2-a896-49b5-ad01-4b07d3861161" containerName="cleanup" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108518 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="902ae94e-7b7f-42ba-a9e3-d8c2db9ba71f" containerName="revoke-keys" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108524 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="52879a72-ff92-4f16-83bd-0f65dd98019d" containerName="cleanup" Nov 02 18:01:04.108642 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.108530 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="0558ce75-6fbc-442b-8024-6b66f00e5f9f" containerName="istio-proxy" Nov 02 18:01:04.111720 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.111703 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:04.116352 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.116333 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:01:04.116545 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.116529 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:01:04.123915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.123890 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4"] Nov 02 18:01:04.207833 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.207794 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:04.208032 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.207982 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4rpz\" (UniqueName: \"kubernetes.io/projected/d2608c5a-1867-452f-a961-dd5975c65bf4-kube-api-access-p4rpz\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:04.208138 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.208046 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:04.309221 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.309176 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4rpz\" (UniqueName: \"kubernetes.io/projected/d2608c5a-1867-452f-a961-dd5975c65bf4-kube-api-access-p4rpz\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:04.309221 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.309229 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:04.309449 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.309290 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:04.310015 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.309983 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:04.311610 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.311592 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:04.319277 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.319256 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4rpz\" (UniqueName: \"kubernetes.io/projected/d2608c5a-1867-452f-a961-dd5975c65bf4-kube-api-access-p4rpz\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:04.426177 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.426073 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:04.620773 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.620743 4310 scope.go:122] "RemoveContainer" containerID="e2a325bda07adb7feb4ebecf1f9d2849a6c2178d053249ac45792f659a3ba6fa" Nov 02 18:01:04.630476 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.630457 4310 scope.go:122] "RemoveContainer" containerID="a25e167564caf64a83ad54b12330bd85872b10f0d24fa83e01ce918f9e83829d" Nov 02 18:01:04.640389 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.640283 4310 scope.go:122] "RemoveContainer" containerID="01d0fad1b830331de8a575812535a8ce477ee5859508638a2d6fa9d3958bf4a4" Nov 02 18:01:04.651611 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.651588 4310 scope.go:122] "RemoveContainer" containerID="cc5062a8107408298db24d535bcc50648d6ef4467d22dbcce38012dfafbd180d" Nov 02 18:01:04.660565 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.660547 4310 scope.go:122] "RemoveContainer" containerID="6b140c474efac7437eceb99948b09b12518b612aa3a3c8d0c606acdca592fc5d" Nov 02 18:01:04.670933 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.670910 4310 scope.go:122] "RemoveContainer" containerID="1cfbdab28178f0f79863a0218e1bd95bd980b935ca6aab8b75cfc5025382f2f0" Nov 02 18:01:04.682019 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.682001 4310 scope.go:122] "RemoveContainer" containerID="a210a0914e9ba37eacb4e0ba7303fca8ff5654f32243190682d286ebacacabd3" Nov 02 18:01:04.692711 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.692688 4310 scope.go:122] "RemoveContainer" containerID="b9da58ac8049143892414e5578bcb4c56ec6e8bafe036d94466118dddd2fbd58" Nov 02 18:01:04.702336 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.702318 4310 scope.go:122] "RemoveContainer" containerID="6bff4cedbf7ed08366da770e026da7b91dba9c390f9c942128346031c28d3f5e" Nov 02 18:01:04.710904 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.710876 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4"] Nov 02 18:01:04.722639 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.722616 4310 scope.go:122] "RemoveContainer" containerID="ad379257f8490a7f232b49f9e6b2fdc72046a4a28c8bf3177c2edcf0ef6f0ddf" Nov 02 18:01:04.736475 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.736454 4310 scope.go:122] "RemoveContainer" containerID="855b95009d09dec8f3e7b549b3b581c02249bc99172be0fa844c7cab672e3437" Nov 02 18:01:04.749605 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.749582 4310 scope.go:122] "RemoveContainer" containerID="98d448aff8b4395afb8b83fc5769c44117acd32c147822ed163ff9a3e6ebecae" Nov 02 18:01:04.760163 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.760137 4310 scope.go:122] "RemoveContainer" containerID="be233ef3a371e4d01f0ccc99f8d2b014d132ede8e369b97705c5c5d1a9361750" Nov 02 18:01:04.760688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.760593 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:04.778100 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.778064 4310 scope.go:122] "RemoveContainer" containerID="80ba3f95e1d4a2b57830444628520e361087f7f72d56be881c301104dfdc83c0" Nov 02 18:01:04.787191 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:04.787171 4310 scope.go:122] "RemoveContainer" containerID="51aba474f2321f5ed5051a22f6a51174d009baac7480b91394c4d4d745e98011" Nov 02 18:01:05.023475 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:05.023438 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6"] Nov 02 18:01:05.026251 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:05.026231 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140daf35-nhsh6"] Nov 02 18:01:05.358293 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:05.358248 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" event={"ID":"d2608c5a-1867-452f-a961-dd5975c65bf4","Type":"ContainerStarted","Data":"fa786e7120ea527679229304598fe9c41557513560937f404c47cc42b545b16a"} Nov 02 18:01:05.358721 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:05.358353 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" event={"ID":"d2608c5a-1867-452f-a961-dd5975c65bf4","Type":"ContainerStarted","Data":"422b147dfaa6cdf2e6e6de480317b92a974221f138d970069e8f200980d03cc3"} Nov 02 18:01:05.376767 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:05.376714 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" podStartSLOduration=1.376699708 podStartE2EDuration="1.376699708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:05.375841486 +0000 UTC m=+1262.818924124" watchObservedRunningTime="2026-11-02 18:01:05.376699708 +0000 UTC m=+1262.819782343" Nov 02 18:01:05.485399 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:05.485361 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:06.365990 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:06.365957 4310 generic.go:356] "Generic (PLEG): container finished" podID="d2608c5a-1867-452f-a961-dd5975c65bf4" containerID="fa786e7120ea527679229304598fe9c41557513560937f404c47cc42b545b16a" exitCode=0 Nov 02 18:01:06.366453 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:06.366044 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" event={"ID":"d2608c5a-1867-452f-a961-dd5975c65bf4","Type":"ContainerDied","Data":"fa786e7120ea527679229304598fe9c41557513560937f404c47cc42b545b16a"} Nov 02 18:01:06.429518 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:06.421490 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:06.672861 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:06.672774 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd6a9c0-5175-4140-9682-6d369c7b7825" path="/var/lib/kubelet/pods/acd6a9c0-5175-4140-9682-6d369c7b7825/volumes" Nov 02 18:01:07.075059 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.075027 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:07.515263 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.515241 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:07.558437 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.558398 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-xks-ca\") pod \"d2608c5a-1867-452f-a961-dd5975c65bf4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " Nov 02 18:01:07.558628 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.558542 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2608c5a-1867-452f-a961-dd5975c65bf4-kube-api-access-p4rpz\" (UniqueName: \"kubernetes.io/projected/d2608c5a-1867-452f-a961-dd5975c65bf4-kube-api-access-p4rpz\") pod \"d2608c5a-1867-452f-a961-dd5975c65bf4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " Nov 02 18:01:07.558628 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.558583 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-openshift-ca\") pod \"d2608c5a-1867-452f-a961-dd5975c65bf4\" (UID: \"d2608c5a-1867-452f-a961-dd5975c65bf4\") " Nov 02 18:01:07.559008 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.558972 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-openshift-ca" pod "d2608c5a-1867-452f-a961-dd5975c65bf4" (UID: "d2608c5a-1867-452f-a961-dd5975c65bf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:07.560565 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.560536 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-xks-ca" pod "d2608c5a-1867-452f-a961-dd5975c65bf4" (UID: "d2608c5a-1867-452f-a961-dd5975c65bf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:07.560722 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.560703 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2608c5a-1867-452f-a961-dd5975c65bf4-kube-api-access-p4rpz" pod "d2608c5a-1867-452f-a961-dd5975c65bf4" (UID: "d2608c5a-1867-452f-a961-dd5975c65bf4"). InnerVolumeSpecName "kube-api-access-p4rpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:07.659833 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.659750 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:07.659833 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.659780 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4rpz\" (UniqueName: \"kubernetes.io/projected/d2608c5a-1867-452f-a961-dd5975c65bf4-kube-api-access-p4rpz\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:07.659833 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.659791 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2608c5a-1867-452f-a961-dd5975c65bf4-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:07.883671 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:07.883638 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:08.380197 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:08.380158 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" event={"ID":"d2608c5a-1867-452f-a961-dd5975c65bf4","Type":"ContainerDied","Data":"422b147dfaa6cdf2e6e6de480317b92a974221f138d970069e8f200980d03cc3"} Nov 02 18:01:08.380197 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:08.380182 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4" Nov 02 18:01:08.380413 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:08.380215 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="422b147dfaa6cdf2e6e6de480317b92a974221f138d970069e8f200980d03cc3" Nov 02 18:01:08.680345 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:08.680225 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:08.906271 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:08.906239 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh"] Nov 02 18:01:08.913918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:08.913890 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs"] Nov 02 18:01:09.032815 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:09.032779 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:09.496762 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:09.496730 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:10.165408 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:10.165372 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:10.937610 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:10.937564 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:11.576952 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:11.576911 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:12.283817 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:12.283787 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:13.069539 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:13.069499 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:13.975241 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:13.975208 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:14.391680 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.391599 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:14.524262 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.524226 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:14.715405 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.715372 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd"] Nov 02 18:01:14.715943 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.715921 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d2608c5a-1867-452f-a961-dd5975c65bf4" containerName="revoke-keys" Nov 02 18:01:14.716030 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.715946 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2608c5a-1867-452f-a961-dd5975c65bf4" containerName="revoke-keys" Nov 02 18:01:14.716110 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.716048 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2608c5a-1867-452f-a961-dd5975c65bf4" containerName="revoke-keys" Nov 02 18:01:14.719360 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.719340 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:14.727843 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.727820 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:01:14.729274 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.729254 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:01:14.732845 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.732820 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr"] Nov 02 18:01:14.736607 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.736585 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd"] Nov 02 18:01:14.736721 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.736675 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:14.755582 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.755543 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr"] Nov 02 18:01:14.838539 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.838508 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpt86\" (UniqueName: \"kubernetes.io/projected/48c9d32a-137e-444b-99a1-bcec773e85d5-kube-api-access-xpt86\") pod \"maas-api-revoke-sub-436ac4ab-dztfr\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:14.838721 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.838612 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436ac4ab-dztfr\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:14.838721 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.838645 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqw2n\" (UniqueName: \"kubernetes.io/projected/357f7f96-2e25-4fb2-b836-7ff645d55283-kube-api-access-lqw2n\") pod \"maas-api-revoke-sub-29b2efea-prmxd\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:14.838811 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.838739 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b2efea-prmxd\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:14.838811 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.838797 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b2efea-prmxd\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:14.838910 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.838895 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436ac4ab-dztfr\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:14.940040 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.939999 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpt86\" (UniqueName: \"kubernetes.io/projected/48c9d32a-137e-444b-99a1-bcec773e85d5-kube-api-access-xpt86\") pod \"maas-api-revoke-sub-436ac4ab-dztfr\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:14.940271 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.940063 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436ac4ab-dztfr\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:14.940271 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.940129 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw2n\" (UniqueName: \"kubernetes.io/projected/357f7f96-2e25-4fb2-b836-7ff645d55283-kube-api-access-lqw2n\") pod \"maas-api-revoke-sub-29b2efea-prmxd\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:14.940271 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.940156 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b2efea-prmxd\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:14.940271 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.940191 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b2efea-prmxd\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:14.940413 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.940346 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436ac4ab-dztfr\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:14.940855 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.940820 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436ac4ab-dztfr\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:14.941006 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.940872 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b2efea-prmxd\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:14.942614 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.942589 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436ac4ab-dztfr\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:14.942614 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.942609 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b2efea-prmxd\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:14.949062 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.949029 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqw2n\" (UniqueName: \"kubernetes.io/projected/357f7f96-2e25-4fb2-b836-7ff645d55283-kube-api-access-lqw2n\") pod \"maas-api-revoke-sub-29b2efea-prmxd\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:14.949578 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:14.949559 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpt86\" (UniqueName: \"kubernetes.io/projected/48c9d32a-137e-444b-99a1-bcec773e85d5-kube-api-access-xpt86\") pod \"maas-api-revoke-sub-436ac4ab-dztfr\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:15.034294 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:15.034207 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:15.053235 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:15.053205 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:15.320425 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:15.320400 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd"] Nov 02 18:01:15.321552 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:01:15.321528 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357f7f96_2e25_4fb2_b836_7ff645d55283.slice/crio-c55a5757c29cedd0fc4779fc1e3dc6c0b1b24051dfa175d86c3f6f2c2479da9d WatchSource:0}: Error finding container c55a5757c29cedd0fc4779fc1e3dc6c0b1b24051dfa175d86c3f6f2c2479da9d: Status 404 returned error can't find the container with id c55a5757c29cedd0fc4779fc1e3dc6c0b1b24051dfa175d86c3f6f2c2479da9d Nov 02 18:01:15.348203 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:15.348172 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr"] Nov 02 18:01:15.349109 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:01:15.349062 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c9d32a_137e_444b_99a1_bcec773e85d5.slice/crio-8627f14b8aae6a424c8ae990142484a35b387efe60a6d3c3b81bbcfd734e8fa0 WatchSource:0}: Error finding container 8627f14b8aae6a424c8ae990142484a35b387efe60a6d3c3b81bbcfd734e8fa0: Status 404 returned error can't find the container with id 8627f14b8aae6a424c8ae990142484a35b387efe60a6d3c3b81bbcfd734e8fa0 Nov 02 18:01:15.422735 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:15.422700 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" event={"ID":"48c9d32a-137e-444b-99a1-bcec773e85d5","Type":"ContainerStarted","Data":"8627f14b8aae6a424c8ae990142484a35b387efe60a6d3c3b81bbcfd734e8fa0"} Nov 02 18:01:15.424165 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:15.424139 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" event={"ID":"357f7f96-2e25-4fb2-b836-7ff645d55283","Type":"ContainerStarted","Data":"c55a5757c29cedd0fc4779fc1e3dc6c0b1b24051dfa175d86c3f6f2c2479da9d"} Nov 02 18:01:16.433212 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:16.433131 4310 generic.go:356] "Generic (PLEG): container finished" podID="357f7f96-2e25-4fb2-b836-7ff645d55283" containerID="3a3038ebdd5e73c11a13490fbf187345ba82a1763a4720c5b07c76eac2357984" exitCode=0 Nov 02 18:01:16.433618 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:16.433257 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" event={"ID":"357f7f96-2e25-4fb2-b836-7ff645d55283","Type":"ContainerDied","Data":"3a3038ebdd5e73c11a13490fbf187345ba82a1763a4720c5b07c76eac2357984"} Nov 02 18:01:16.434938 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:16.434913 4310 generic.go:356] "Generic (PLEG): container finished" podID="48c9d32a-137e-444b-99a1-bcec773e85d5" containerID="33b79af25383f5255cd636e9957ddd81513fb223457e15ae2967d8df12a57699" exitCode=0 Nov 02 18:01:16.435054 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:16.434989 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" event={"ID":"48c9d32a-137e-444b-99a1-bcec773e85d5","Type":"ContainerDied","Data":"33b79af25383f5255cd636e9957ddd81513fb223457e15ae2967d8df12a57699"} Nov 02 18:01:17.037958 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.037922 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:17.156234 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.156197 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:17.607878 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.607856 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:17.610929 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.610913 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:17.667691 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.667652 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c9d32a-137e-444b-99a1-bcec773e85d5-kube-api-access-xpt86\" (UniqueName: \"kubernetes.io/projected/48c9d32a-137e-444b-99a1-bcec773e85d5-kube-api-access-xpt86\") pod \"48c9d32a-137e-444b-99a1-bcec773e85d5\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " Nov 02 18:01:17.667880 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.667709 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357f7f96-2e25-4fb2-b836-7ff645d55283-kube-api-access-lqw2n\" (UniqueName: \"kubernetes.io/projected/357f7f96-2e25-4fb2-b836-7ff645d55283-kube-api-access-lqw2n\") pod \"357f7f96-2e25-4fb2-b836-7ff645d55283\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " Nov 02 18:01:17.667880 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.667743 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-openshift-ca\") pod \"357f7f96-2e25-4fb2-b836-7ff645d55283\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " Nov 02 18:01:17.668004 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.667875 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-openshift-ca\") pod \"48c9d32a-137e-444b-99a1-bcec773e85d5\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " Nov 02 18:01:17.668004 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.667931 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-xks-ca\") pod \"357f7f96-2e25-4fb2-b836-7ff645d55283\" (UID: \"357f7f96-2e25-4fb2-b836-7ff645d55283\") " Nov 02 18:01:17.668004 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.667982 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-xks-ca\") pod \"48c9d32a-137e-444b-99a1-bcec773e85d5\" (UID: \"48c9d32a-137e-444b-99a1-bcec773e85d5\") " Nov 02 18:01:17.668320 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.668291 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-openshift-ca" pod "357f7f96-2e25-4fb2-b836-7ff645d55283" (UID: "357f7f96-2e25-4fb2-b836-7ff645d55283"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:17.668410 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.668384 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-openshift-ca" pod "48c9d32a-137e-444b-99a1-bcec773e85d5" (UID: "48c9d32a-137e-444b-99a1-bcec773e85d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:17.668473 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.668455 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:17.671205 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.670877 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357f7f96-2e25-4fb2-b836-7ff645d55283-kube-api-access-lqw2n" pod "357f7f96-2e25-4fb2-b836-7ff645d55283" (UID: "357f7f96-2e25-4fb2-b836-7ff645d55283"). InnerVolumeSpecName "kube-api-access-lqw2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:17.671342 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.671321 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c9d32a-137e-444b-99a1-bcec773e85d5-kube-api-access-xpt86" pod "48c9d32a-137e-444b-99a1-bcec773e85d5" (UID: "48c9d32a-137e-444b-99a1-bcec773e85d5"). InnerVolumeSpecName "kube-api-access-xpt86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:17.671503 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.671470 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-xks-ca" pod "357f7f96-2e25-4fb2-b836-7ff645d55283" (UID: "357f7f96-2e25-4fb2-b836-7ff645d55283"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:17.675220 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.675194 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-xks-ca" pod "48c9d32a-137e-444b-99a1-bcec773e85d5" (UID: "48c9d32a-137e-444b-99a1-bcec773e85d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:17.769550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.769516 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:17.769550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.769546 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpt86\" (UniqueName: \"kubernetes.io/projected/48c9d32a-137e-444b-99a1-bcec773e85d5-kube-api-access-xpt86\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:17.769550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.769555 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqw2n\" (UniqueName: \"kubernetes.io/projected/357f7f96-2e25-4fb2-b836-7ff645d55283-kube-api-access-lqw2n\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:17.769783 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.769564 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c9d32a-137e-444b-99a1-bcec773e85d5-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:17.769783 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:17.769574 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/357f7f96-2e25-4fb2-b836-7ff645d55283-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:18.449125 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:18.449092 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" Nov 02 18:01:18.449294 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:18.449114 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr" event={"ID":"48c9d32a-137e-444b-99a1-bcec773e85d5","Type":"ContainerDied","Data":"8627f14b8aae6a424c8ae990142484a35b387efe60a6d3c3b81bbcfd734e8fa0"} Nov 02 18:01:18.449294 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:18.449177 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8627f14b8aae6a424c8ae990142484a35b387efe60a6d3c3b81bbcfd734e8fa0" Nov 02 18:01:18.450690 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:18.450670 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" Nov 02 18:01:18.450813 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:18.450694 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd" event={"ID":"357f7f96-2e25-4fb2-b836-7ff645d55283","Type":"ContainerDied","Data":"c55a5757c29cedd0fc4779fc1e3dc6c0b1b24051dfa175d86c3f6f2c2479da9d"} Nov 02 18:01:18.450813 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:18.450727 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c55a5757c29cedd0fc4779fc1e3dc6c0b1b24051dfa175d86c3f6f2c2479da9d" Nov 02 18:01:23.928251 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:23.928197 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" containerID="cri-o://4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055" gracePeriod=45 Nov 02 18:01:23.937581 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:23.937529 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" containerID="cri-o://db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f" gracePeriod=45 Nov 02 18:01:24.188364 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.188297 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 18:01:24.221618 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.221589 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 18:01:24.341749 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.341716 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85ade69-b5f2-4498-a3de-2a1d8a084230-tls-certs\" (UniqueName: \"kubernetes.io/secret/d85ade69-b5f2-4498-a3de-2a1d8a084230-tls-certs\") pod \"d85ade69-b5f2-4498-a3de-2a1d8a084230\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " Nov 02 18:01:24.341915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.341779 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-home\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-home\") pod \"dbd26314-4dca-4ac0-9cee-25efded09e70\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " Nov 02 18:01:24.341915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.341797 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbd26314-4dca-4ac0-9cee-25efded09e70-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd26314-4dca-4ac0-9cee-25efded09e70-tls-certs\") pod \"dbd26314-4dca-4ac0-9cee-25efded09e70\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " Nov 02 18:01:24.341915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.341826 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-home\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-home\") pod \"d85ade69-b5f2-4498-a3de-2a1d8a084230\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " Nov 02 18:01:24.341915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.341851 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85ade69-b5f2-4498-a3de-2a1d8a084230-kube-api-access-rmbh9\" (UniqueName: \"kubernetes.io/projected/d85ade69-b5f2-4498-a3de-2a1d8a084230-kube-api-access-rmbh9\") pod \"d85ade69-b5f2-4498-a3de-2a1d8a084230\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " Nov 02 18:01:24.341915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.341871 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-tmp-dir\") pod \"dbd26314-4dca-4ac0-9cee-25efded09e70\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " Nov 02 18:01:24.341915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.341908 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-dshm\") pod \"dbd26314-4dca-4ac0-9cee-25efded09e70\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " Nov 02 18:01:24.342349 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.341963 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-model-cache\") pod \"dbd26314-4dca-4ac0-9cee-25efded09e70\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " Nov 02 18:01:24.342349 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342013 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-dshm\") pod \"d85ade69-b5f2-4498-a3de-2a1d8a084230\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " Nov 02 18:01:24.342349 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342051 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-tmp-dir\") pod \"d85ade69-b5f2-4498-a3de-2a1d8a084230\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " Nov 02 18:01:24.342349 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342113 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-model-cache\") pod \"d85ade69-b5f2-4498-a3de-2a1d8a084230\" (UID: \"d85ade69-b5f2-4498-a3de-2a1d8a084230\") " Nov 02 18:01:24.342349 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342149 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd26314-4dca-4ac0-9cee-25efded09e70-kube-api-access-dbn2s\" (UniqueName: \"kubernetes.io/projected/dbd26314-4dca-4ac0-9cee-25efded09e70-kube-api-access-dbn2s\") pod \"dbd26314-4dca-4ac0-9cee-25efded09e70\" (UID: \"dbd26314-4dca-4ac0-9cee-25efded09e70\") " Nov 02 18:01:24.342349 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342166 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-home" pod "dbd26314-4dca-4ac0-9cee-25efded09e70" (UID: "dbd26314-4dca-4ac0-9cee-25efded09e70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.342349 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342180 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-tmp-dir" pod "dbd26314-4dca-4ac0-9cee-25efded09e70" (UID: "dbd26314-4dca-4ac0-9cee-25efded09e70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.342349 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342348 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-home" pod "d85ade69-b5f2-4498-a3de-2a1d8a084230" (UID: "d85ade69-b5f2-4498-a3de-2a1d8a084230"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.342808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342388 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-model-cache" pod "dbd26314-4dca-4ac0-9cee-25efded09e70" (UID: "dbd26314-4dca-4ac0-9cee-25efded09e70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.342808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342551 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-model-cache" pod "d85ade69-b5f2-4498-a3de-2a1d8a084230" (UID: "d85ade69-b5f2-4498-a3de-2a1d8a084230"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.342808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342731 4310 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-model-cache\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.342808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342751 4310 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-model-cache\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.342808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342757 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-tmp-dir" pod "d85ade69-b5f2-4498-a3de-2a1d8a084230" (UID: "d85ade69-b5f2-4498-a3de-2a1d8a084230"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.342808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342765 4310 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-home\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.342808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342798 4310 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-home\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.343233 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.342814 4310 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-tmp-dir\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.344508 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.344482 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-dshm" pod "d85ade69-b5f2-4498-a3de-2a1d8a084230" (UID: "d85ade69-b5f2-4498-a3de-2a1d8a084230"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.344508 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.344495 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd26314-4dca-4ac0-9cee-25efded09e70-kube-api-access-dbn2s" pod "dbd26314-4dca-4ac0-9cee-25efded09e70" (UID: "dbd26314-4dca-4ac0-9cee-25efded09e70"). InnerVolumeSpecName "kube-api-access-dbn2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:24.344673 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.344556 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-dshm" pod "dbd26314-4dca-4ac0-9cee-25efded09e70" (UID: "dbd26314-4dca-4ac0-9cee-25efded09e70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:24.344673 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.344604 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd26314-4dca-4ac0-9cee-25efded09e70-tls-certs" pod "dbd26314-4dca-4ac0-9cee-25efded09e70" (UID: "dbd26314-4dca-4ac0-9cee-25efded09e70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:24.344673 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.344612 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85ade69-b5f2-4498-a3de-2a1d8a084230-tls-certs" pod "d85ade69-b5f2-4498-a3de-2a1d8a084230" (UID: "d85ade69-b5f2-4498-a3de-2a1d8a084230"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:24.344867 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.344841 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85ade69-b5f2-4498-a3de-2a1d8a084230-kube-api-access-rmbh9" pod "d85ade69-b5f2-4498-a3de-2a1d8a084230" (UID: "d85ade69-b5f2-4498-a3de-2a1d8a084230"). InnerVolumeSpecName "kube-api-access-rmbh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:24.443467 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.443378 4310 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd26314-4dca-4ac0-9cee-25efded09e70-tls-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.443467 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.443412 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmbh9\" (UniqueName: \"kubernetes.io/projected/d85ade69-b5f2-4498-a3de-2a1d8a084230-kube-api-access-rmbh9\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.443467 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.443421 4310 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd26314-4dca-4ac0-9cee-25efded09e70-dshm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.443467 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.443430 4310 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-dshm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.443467 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.443438 4310 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85ade69-b5f2-4498-a3de-2a1d8a084230-tmp-dir\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.443467 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.443445 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbn2s\" (UniqueName: \"kubernetes.io/projected/dbd26314-4dca-4ac0-9cee-25efded09e70-kube-api-access-dbn2s\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.443467 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.443453 4310 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85ade69-b5f2-4498-a3de-2a1d8a084230-tls-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:24.487139 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.487107 4310 generic.go:356] "Generic (PLEG): container finished" podID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerID="4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055" exitCode=0 Nov 02 18:01:24.487337 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.487156 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" Nov 02 18:01:24.487337 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.487196 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" event={"ID":"dbd26314-4dca-4ac0-9cee-25efded09e70","Type":"ContainerDied","Data":"4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055"} Nov 02 18:01:24.487337 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.487244 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh" event={"ID":"dbd26314-4dca-4ac0-9cee-25efded09e70","Type":"ContainerDied","Data":"7897f1183a7246caa14c0da6c2b714c5fa3cf33da2116a1ef70e575cf475eabb"} Nov 02 18:01:24.487337 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.487266 4310 scope.go:122] "RemoveContainer" containerID="4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055" Nov 02 18:01:24.488565 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.488543 4310 generic.go:356] "Generic (PLEG): container finished" podID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerID="db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f" exitCode=0 Nov 02 18:01:24.488679 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.488663 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" event={"ID":"d85ade69-b5f2-4498-a3de-2a1d8a084230","Type":"ContainerDied","Data":"db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f"} Nov 02 18:01:24.488729 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.488700 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" event={"ID":"d85ade69-b5f2-4498-a3de-2a1d8a084230","Type":"ContainerDied","Data":"2ff162a322c0f4dce90accaf8fabba271954e731778d80f8b287023eea799f7d"} Nov 02 18:01:24.488769 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.488760 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs" Nov 02 18:01:24.502190 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.502170 4310 scope.go:122] "RemoveContainer" containerID="4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055" Nov 02 18:01:24.502525 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:01:24.502498 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055\": container with ID starting with 4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055 not found: ID does not exist" containerID="4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055" Nov 02 18:01:24.502603 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.502528 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055"} err="failed to get container status \"4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055\": rpc error: code = NotFound desc = could not find container \"4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055\": container with ID starting with 4afee9d86a35901a95aa12b1998721396c8bd94cd027f23e25d24e34e575c055 not found: ID does not exist" Nov 02 18:01:24.502603 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.502546 4310 scope.go:122] "RemoveContainer" containerID="db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f" Nov 02 18:01:24.511569 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.511551 4310 scope.go:122] "RemoveContainer" containerID="db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f" Nov 02 18:01:24.511844 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:01:24.511826 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f\": container with ID starting with db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f not found: ID does not exist" containerID="db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f" Nov 02 18:01:24.511892 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.511852 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f"} err="failed to get container status \"db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f\": rpc error: code = NotFound desc = could not find container \"db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f\": container with ID starting with db24e2e67a7b40838cb2389677f80182a9b7a5b00ba46a13e1f898afff09823f not found: ID does not exist" Nov 02 18:01:24.518416 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.518391 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh"] Nov 02 18:01:24.528503 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.528480 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-2-w2-bb4ecb-kserve-cc84f85f4-h45nh"] Nov 02 18:01:24.545155 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.545125 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs"] Nov 02 18:01:24.549877 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.549850 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb4ecb/distinct-w2-bb4ecb-kserve-6585cdf49-9bhhs"] Nov 02 18:01:24.671813 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.671781 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" path="/var/lib/kubelet/pods/d85ade69-b5f2-4498-a3de-2a1d8a084230/volumes" Nov 02 18:01:24.672237 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:24.672223 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" path="/var/lib/kubelet/pods/dbd26314-4dca-4ac0-9cee-25efded09e70/volumes" Nov 02 18:01:26.758508 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:26.758471 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6"] Nov 02 18:01:26.758931 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:26.758653 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" podUID="30c85a7c-1f56-4fe5-aba6-28408aaacc9c" containerName="discovery" containerID="cri-o://8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c" gracePeriod=30 Nov 02 18:01:27.019309 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.019248 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 18:01:27.168581 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.168546 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-local-certs\") pod \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " Nov 02 18:01:27.168784 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.168590 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-cacerts\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-cacerts\") pod \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " Nov 02 18:01:27.168784 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.168633 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-dns-cert\") pod \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " Nov 02 18:01:27.168784 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.168681 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-kube-api-access-jmdjr\" (UniqueName: \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-kube-api-access-jmdjr\") pod \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " Nov 02 18:01:27.168784 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.168716 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-kubeconfig\") pod \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " Nov 02 18:01:27.168784 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.168769 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-token\" (UniqueName: \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-token\") pod \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " Nov 02 18:01:27.169122 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.168852 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-ca-configmap\") pod \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\" (UID: \"30c85a7c-1f56-4fe5-aba6-28408aaacc9c\") " Nov 02 18:01:27.169497 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.169463 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-ca-configmap" pod "30c85a7c-1f56-4fe5-aba6-28408aaacc9c" (UID: "30c85a7c-1f56-4fe5-aba6-28408aaacc9c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:27.170926 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.170883 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-cacerts" pod "30c85a7c-1f56-4fe5-aba6-28408aaacc9c" (UID: "30c85a7c-1f56-4fe5-aba6-28408aaacc9c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:27.171115 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.171064 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-kubeconfig" pod "30c85a7c-1f56-4fe5-aba6-28408aaacc9c" (UID: "30c85a7c-1f56-4fe5-aba6-28408aaacc9c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:27.171291 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.171266 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-local-certs" pod "30c85a7c-1f56-4fe5-aba6-28408aaacc9c" (UID: "30c85a7c-1f56-4fe5-aba6-28408aaacc9c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:27.171371 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.171328 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-kube-api-access-jmdjr" pod "30c85a7c-1f56-4fe5-aba6-28408aaacc9c" (UID: "30c85a7c-1f56-4fe5-aba6-28408aaacc9c"). InnerVolumeSpecName "kube-api-access-jmdjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:27.171490 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.171473 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-dns-cert" pod "30c85a7c-1f56-4fe5-aba6-28408aaacc9c" (UID: "30c85a7c-1f56-4fe5-aba6-28408aaacc9c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:27.171531 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.171487 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-token" pod "30c85a7c-1f56-4fe5-aba6-28408aaacc9c" (UID: "30c85a7c-1f56-4fe5-aba6-28408aaacc9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:27.270535 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.270450 4310 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-ca-configmap\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:27.270535 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.270485 4310 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-local-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:27.270535 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.270494 4310 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-cacerts\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:27.270535 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.270505 4310 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-csr-dns-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:27.270535 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.270514 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmdjr\" (UniqueName: \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-kube-api-access-jmdjr\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:27.270535 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.270522 4310 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-kubeconfig\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:27.270535 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.270532 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c85a7c-1f56-4fe5-aba6-28408aaacc9c-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:27.510092 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.510040 4310 generic.go:356] "Generic (PLEG): container finished" podID="30c85a7c-1f56-4fe5-aba6-28408aaacc9c" containerID="8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c" exitCode=0 Nov 02 18:01:27.510265 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.510115 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" Nov 02 18:01:27.510265 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.510127 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" event={"ID":"30c85a7c-1f56-4fe5-aba6-28408aaacc9c","Type":"ContainerDied","Data":"8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c"} Nov 02 18:01:27.510265 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.510185 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6" event={"ID":"30c85a7c-1f56-4fe5-aba6-28408aaacc9c","Type":"ContainerDied","Data":"74e980116b31d90054ea394d9f32c89a713b9c28efdf89753e0c88c518e24b3e"} Nov 02 18:01:27.510265 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.510208 4310 scope.go:122] "RemoveContainer" containerID="8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c" Nov 02 18:01:27.522715 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.522693 4310 scope.go:122] "RemoveContainer" containerID="8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c" Nov 02 18:01:27.523047 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:01:27.523022 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c\": container with ID starting with 8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c not found: ID does not exist" containerID="8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c" Nov 02 18:01:27.523209 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.523054 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c"} err="failed to get container status \"8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c\": rpc error: code = NotFound desc = could not find container \"8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c\": container with ID starting with 8197c1dd2b4a8bf3faae380add59d32fc16db259d49d25c60811ca59af83304c not found: ID does not exist" Nov 02 18:01:27.550865 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.550829 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6"] Nov 02 18:01:27.558957 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:27.558920 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmfb6"] Nov 02 18:01:28.673148 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:28.673115 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c85a7c-1f56-4fe5-aba6-28408aaacc9c" path="/var/lib/kubelet/pods/30c85a7c-1f56-4fe5-aba6-28408aaacc9c/volumes" Nov 02 18:01:29.899538 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:29.899499 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:30.057517 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:30.057478 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:30.072611 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:30.072579 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4"] Nov 02 18:01:30.080503 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:30.080470 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb4ecb-bbfh4"] Nov 02 18:01:30.177055 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:30.176981 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:30.644174 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:30.644137 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:30.677898 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:30.677857 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2608c5a-1867-452f-a961-dd5975c65bf4" path="/var/lib/kubelet/pods/d2608c5a-1867-452f-a961-dd5975c65bf4/volumes" Nov 02 18:01:33.541232 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:01:33.541195 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:01:36.982040 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:36.982005 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:37.088602 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:37.088574 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:37.353227 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:37.352098 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5"] Nov 02 18:01:37.353227 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:37.352325 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" containerID="cri-o://cfc57f5019d0505b8ec132bf473a754b51a4a9420410d89f4054c93784c47b4a" gracePeriod=30 Nov 02 18:01:37.422645 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:37.422611 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:39.989806 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:39.989759 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-698c564c8b-v5qgk"] Nov 02 18:01:39.990339 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:39.989948 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-698c564c8b-v5qgk" podUID="d46ea576-9209-41ed-96c0-9ced3e50d3ae" containerName="manager" containerID="cri-o://b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6" gracePeriod=30 Nov 02 18:01:40.243672 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.243613 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 18:01:40.308712 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.308681 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-webhook-cert\") pod \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " Nov 02 18:01:40.308909 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.308802 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-metrics-certs\") pod \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " Nov 02 18:01:40.308909 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.308837 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46ea576-9209-41ed-96c0-9ced3e50d3ae-kube-api-access-8twv2\" (UniqueName: \"kubernetes.io/projected/d46ea576-9209-41ed-96c0-9ced3e50d3ae-kube-api-access-8twv2\") pod \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\" (UID: \"d46ea576-9209-41ed-96c0-9ced3e50d3ae\") " Nov 02 18:01:40.317544 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.317394 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46ea576-9209-41ed-96c0-9ced3e50d3ae-kube-api-access-8twv2" pod "d46ea576-9209-41ed-96c0-9ced3e50d3ae" (UID: "d46ea576-9209-41ed-96c0-9ced3e50d3ae"). InnerVolumeSpecName "kube-api-access-8twv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:40.321819 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.321783 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-metrics-certs" pod "d46ea576-9209-41ed-96c0-9ced3e50d3ae" (UID: "d46ea576-9209-41ed-96c0-9ced3e50d3ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:40.324576 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.324525 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-webhook-cert" pod "d46ea576-9209-41ed-96c0-9ced3e50d3ae" (UID: "d46ea576-9209-41ed-96c0-9ced3e50d3ae"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:01:40.410827 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.410790 4310 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-metrics-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:40.410827 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.410820 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twv2\" (UniqueName: \"kubernetes.io/projected/d46ea576-9209-41ed-96c0-9ced3e50d3ae-kube-api-access-8twv2\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:40.410827 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.410830 4310 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d46ea576-9209-41ed-96c0-9ced3e50d3ae-webhook-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:40.589924 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.589832 4310 generic.go:356] "Generic (PLEG): container finished" podID="d46ea576-9209-41ed-96c0-9ced3e50d3ae" containerID="b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6" exitCode=0 Nov 02 18:01:40.589924 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.589894 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-698c564c8b-v5qgk" Nov 02 18:01:40.590133 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.589934 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-698c564c8b-v5qgk" event={"ID":"d46ea576-9209-41ed-96c0-9ced3e50d3ae","Type":"ContainerDied","Data":"b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6"} Nov 02 18:01:40.590133 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.590014 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-698c564c8b-v5qgk" event={"ID":"d46ea576-9209-41ed-96c0-9ced3e50d3ae","Type":"ContainerDied","Data":"1ee18a6b544a9615ebc3af783a41009b88504f59542fa95d4e0a713f6a1363eb"} Nov 02 18:01:40.590133 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.590037 4310 scope.go:122] "RemoveContainer" containerID="b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6" Nov 02 18:01:40.600637 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.600617 4310 scope.go:122] "RemoveContainer" containerID="b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6" Nov 02 18:01:40.600915 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:01:40.600897 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6\": container with ID starting with b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6 not found: ID does not exist" containerID="b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6" Nov 02 18:01:40.600979 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.600924 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6"} err="failed to get container status \"b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6\": rpc error: code = NotFound desc = could not find container \"b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6\": container with ID starting with b0a63a246f413a13779e045b8a877e12ef1d43f044308036fc1a6f2b255b72b6 not found: ID does not exist" Nov 02 18:01:40.613023 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.612998 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-698c564c8b-v5qgk"] Nov 02 18:01:40.616776 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.616750 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-698c564c8b-v5qgk"] Nov 02 18:01:40.673906 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:40.673868 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46ea576-9209-41ed-96c0-9ced3e50d3ae" path="/var/lib/kubelet/pods/d46ea576-9209-41ed-96c0-9ced3e50d3ae/volumes" Nov 02 18:01:41.306504 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306470 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-698c564c8b-fvvw4"] Nov 02 18:01:41.306894 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306861 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" Nov 02 18:01:41.306894 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306872 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" Nov 02 18:01:41.306894 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306884 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d46ea576-9209-41ed-96c0-9ced3e50d3ae" containerName="manager" Nov 02 18:01:41.306894 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306889 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46ea576-9209-41ed-96c0-9ced3e50d3ae" containerName="manager" Nov 02 18:01:41.306894 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306896 4310 cpu_manager.go:414] "containerMap: removing container" podUID="357f7f96-2e25-4fb2-b836-7ff645d55283" containerName="revoke-keys" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306902 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357f7f96-2e25-4fb2-b836-7ff645d55283" containerName="revoke-keys" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306918 4310 cpu_manager.go:414] "containerMap: removing container" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306922 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306933 4310 cpu_manager.go:414] "containerMap: removing container" podUID="30c85a7c-1f56-4fe5-aba6-28408aaacc9c" containerName="discovery" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306938 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c85a7c-1f56-4fe5-aba6-28408aaacc9c" containerName="discovery" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306956 4310 cpu_manager.go:414] "containerMap: removing container" podUID="48c9d32a-137e-444b-99a1-bcec773e85d5" containerName="revoke-keys" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.306960 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c9d32a-137e-444b-99a1-bcec773e85d5" containerName="revoke-keys" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.307012 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c85a7c-1f56-4fe5-aba6-28408aaacc9c" containerName="discovery" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.307021 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c9d32a-137e-444b-99a1-bcec773e85d5" containerName="revoke-keys" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.307028 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85ade69-b5f2-4498-a3de-2a1d8a084230" containerName="main" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.307033 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46ea576-9209-41ed-96c0-9ced3e50d3ae" containerName="manager" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.307039 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="357f7f96-2e25-4fb2-b836-7ff645d55283" containerName="revoke-keys" Nov 02 18:01:41.307053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.307045 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd26314-4dca-4ac0-9cee-25efded09e70" containerName="main" Nov 02 18:01:41.311572 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.311551 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:41.317246 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.317222 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5xmtd\"" Nov 02 18:01:41.320238 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.320217 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 18:01:41.321385 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.321363 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 18:01:41.321493 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.321436 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 18:01:41.325561 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.325534 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-698c564c8b-fvvw4"] Nov 02 18:01:41.419730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.419682 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-webhook-cert\") pod \"maas-controller-698c564c8b-fvvw4\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:41.419915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.419743 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwsz6\" (UniqueName: \"kubernetes.io/projected/75e2fad2-7cea-4b95-ac23-3d98589b105b-kube-api-access-rwsz6\") pod \"maas-controller-698c564c8b-fvvw4\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:41.419915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.419880 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-metrics-certs\") pod \"maas-controller-698c564c8b-fvvw4\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:41.527765 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.527713 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-webhook-cert\") pod \"maas-controller-698c564c8b-fvvw4\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:41.528026 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.527786 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwsz6\" (UniqueName: \"kubernetes.io/projected/75e2fad2-7cea-4b95-ac23-3d98589b105b-kube-api-access-rwsz6\") pod \"maas-controller-698c564c8b-fvvw4\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:41.528026 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.527907 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-metrics-certs\") pod \"maas-controller-698c564c8b-fvvw4\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:41.530889 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.530830 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-webhook-cert\") pod \"maas-controller-698c564c8b-fvvw4\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:41.531127 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.530878 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-metrics-certs\") pod \"maas-controller-698c564c8b-fvvw4\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:41.538184 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.538159 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwsz6\" (UniqueName: \"kubernetes.io/projected/75e2fad2-7cea-4b95-ac23-3d98589b105b-kube-api-access-rwsz6\") pod \"maas-controller-698c564c8b-fvvw4\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:41.625825 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.625711 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:41.949202 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.949172 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-698c564c8b-fvvw4"] Nov 02 18:01:41.950973 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:01:41.950935 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e2fad2_7cea_4b95_ac23_3d98589b105b.slice/crio-c81e78f09d7e5365425e3adc84b44e0f7855a9b3f0c7a6ce4471c610dc9b438a WatchSource:0}: Error finding container c81e78f09d7e5365425e3adc84b44e0f7855a9b3f0c7a6ce4471c610dc9b438a: Status 404 returned error can't find the container with id c81e78f09d7e5365425e3adc84b44e0f7855a9b3f0c7a6ce4471c610dc9b438a Nov 02 18:01:41.952916 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:41.952895 4310 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:01:42.635704 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:42.635668 4310 generic.go:356] "Generic (PLEG): container finished" podID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerID="cfc57f5019d0505b8ec132bf473a754b51a4a9420410d89f4054c93784c47b4a" exitCode=0 Nov 02 18:01:42.636212 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:42.635788 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" event={"ID":"4447e85f-4043-4e1b-b4ab-e9f36fd1438c","Type":"ContainerDied","Data":"cfc57f5019d0505b8ec132bf473a754b51a4a9420410d89f4054c93784c47b4a"} Nov 02 18:01:42.637481 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:42.637457 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-698c564c8b-fvvw4" event={"ID":"75e2fad2-7cea-4b95-ac23-3d98589b105b","Type":"ContainerStarted","Data":"c81e78f09d7e5365425e3adc84b44e0f7855a9b3f0c7a6ce4471c610dc9b438a"} Nov 02 18:01:43.029968 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.029945 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.086991 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-certs\") pod \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.087319 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-certs" pod "4447e85f-4043-4e1b-b4ab-e9f36fd1438c" (UID: "4447e85f-4043-4e1b-b4ab-e9f36fd1438c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.087406 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-podinfo\") pod \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.087564 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-token\" (UniqueName: \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-token\") pod \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.087978 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istiod-ca-cert\") pod \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.088596 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istiod-ca-cert" pod "4447e85f-4043-4e1b-b4ab-e9f36fd1438c" (UID: "4447e85f-4043-4e1b-b4ab-e9f36fd1438c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.088755 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-kube-api-access-6zfj2\" (UniqueName: \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-kube-api-access-6zfj2\") pod \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.089127 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-credential-socket\") pod \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.089173 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-data\") pod \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.089251 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-envoy\") pod \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.089278 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-socket\") pod \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\" (UID: \"4447e85f-4043-4e1b-b4ab-e9f36fd1438c\") " Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.089762 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.091103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.089785 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.094637 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.094164 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-credential-socket" pod "4447e85f-4043-4e1b-b4ab-e9f36fd1438c" (UID: "4447e85f-4043-4e1b-b4ab-e9f36fd1438c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.094637 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.094384 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-data" pod "4447e85f-4043-4e1b-b4ab-e9f36fd1438c" (UID: "4447e85f-4043-4e1b-b4ab-e9f36fd1438c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.110595 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.110565 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-socket" pod "4447e85f-4043-4e1b-b4ab-e9f36fd1438c" (UID: "4447e85f-4043-4e1b-b4ab-e9f36fd1438c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.121330 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.121280 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-token" pod "4447e85f-4043-4e1b-b4ab-e9f36fd1438c" (UID: "4447e85f-4043-4e1b-b4ab-e9f36fd1438c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:43.121603 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.121579 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-podinfo" pod "4447e85f-4043-4e1b-b4ab-e9f36fd1438c" (UID: "4447e85f-4043-4e1b-b4ab-e9f36fd1438c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:01:43.124259 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.124219 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-envoy" pod "4447e85f-4043-4e1b-b4ab-e9f36fd1438c" (UID: "4447e85f-4043-4e1b-b4ab-e9f36fd1438c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:01:43.124663 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.124628 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-kube-api-access-6zfj2" pod "4447e85f-4043-4e1b-b4ab-e9f36fd1438c" (UID: "4447e85f-4043-4e1b-b4ab-e9f36fd1438c"). InnerVolumeSpecName "kube-api-access-6zfj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:43.196157 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.192701 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.196157 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.192742 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.196157 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.192759 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zfj2\" (UniqueName: \"kubernetes.io/projected/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-kube-api-access-6zfj2\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.196157 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.192775 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.196157 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.192789 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.196157 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.192803 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.196157 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.192818 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e85f-4043-4e1b-b4ab-e9f36fd1438c-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:01:43.661491 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.661450 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" event={"ID":"4447e85f-4043-4e1b-b4ab-e9f36fd1438c","Type":"ContainerDied","Data":"0b5a3fff6941adcfe12bccf5c548c57091fc59544b300eaad9a3566fb7cad0cf"} Nov 02 18:01:43.662142 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.662118 4310 scope.go:122] "RemoveContainer" containerID="cfc57f5019d0505b8ec132bf473a754b51a4a9420410d89f4054c93784c47b4a" Nov 02 18:01:43.662471 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.662451 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5" Nov 02 18:01:43.692106 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.690441 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-698c564c8b-fvvw4" event={"ID":"75e2fad2-7cea-4b95-ac23-3d98589b105b","Type":"ContainerStarted","Data":"0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93"} Nov 02 18:01:43.692106 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.691143 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:43.735538 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.735476 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-698c564c8b-fvvw4" podStartSLOduration=2.210769263 podStartE2EDuration="2.735456046s" totalImagesPullingTime="524.686783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:43.725993911 +0000 UTC m=+1301.169076547" watchObservedRunningTime="2026-11-02 18:01:43.735456046 +0000 UTC m=+1301.178538682" Nov 02 18:01:43.765666 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.765599 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5"] Nov 02 18:01:43.794103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:43.770509 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb4ecb-openshift-default-f98db5885-xprm5"] Nov 02 18:01:44.152317 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:44.152274 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:01:44.152532 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:44.152346 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:01:44.672228 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:44.672197 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" path="/var/lib/kubelet/pods/4447e85f-4043-4e1b-b4ab-e9f36fd1438c/volumes" Nov 02 18:01:46.450697 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:46.450619 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:47.428447 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:47.428412 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:47.635688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:47.635656 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:47.773397 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:47.773362 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:48.092174 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:48.092049 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:48.203417 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:48.203390 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:48.429323 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:48.429243 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:54.020942 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:54.020909 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv"] Nov 02 18:01:54.025074 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:54.025050 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05fbb349-fqhfv"] Nov 02 18:01:54.198834 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:54.198802 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:01:54.673230 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:54.673197 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f80910-45f6-436c-b553-a41ebe2808ea" path="/var/lib/kubelet/pods/b6f80910-45f6-436c-b553-a41ebe2808ea/volumes" Nov 02 18:01:55.035103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:55.035001 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77"] Nov 02 18:01:55.036001 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:55.035975 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23336a10-v5k77"] Nov 02 18:01:55.706400 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:55.706371 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:01:56.672822 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:56.672783 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1183c12f-ff3c-4f0c-adf0-a0883c99e839" path="/var/lib/kubelet/pods/1183c12f-ff3c-4f0c-adf0-a0883c99e839/volumes" Nov 02 18:01:58.024211 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:58.024177 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq"] Nov 02 18:01:58.028668 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:58.028637 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6"] Nov 02 18:01:58.030693 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:58.030665 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b552ade7-lc5lq"] Nov 02 18:01:58.032419 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:58.032396 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9ed352-c9lj6"] Nov 02 18:01:58.673571 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:58.673537 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc14d78-414c-48da-8d5e-0828ea8cc5c1" path="/var/lib/kubelet/pods/3fc14d78-414c-48da-8d5e-0828ea8cc5c1/volumes" Nov 02 18:01:58.673967 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:01:58.673952 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="688fa1f2-1628-404a-9a23-39640c88484c" path="/var/lib/kubelet/pods/688fa1f2-1628-404a-9a23-39640c88484c/volumes" Nov 02 18:02:03.102573 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.102536 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7"] Nov 02 18:02:03.103183 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.103164 4310 cpu_manager.go:414] "containerMap: removing container" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" Nov 02 18:02:03.103236 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.103183 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" Nov 02 18:02:03.103271 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.103257 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="4447e85f-4043-4e1b-b4ab-e9f36fd1438c" containerName="istio-proxy" Nov 02 18:02:03.108002 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.107982 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:03.112350 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.112325 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:02:03.115302 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.115285 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:02:03.118916 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.118890 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7"] Nov 02 18:02:03.214947 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.214908 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xbdk7\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:03.215149 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.214970 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xbdk7\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:03.215149 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.215068 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c2rs\" (UniqueName: \"kubernetes.io/projected/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-kube-api-access-7c2rs\") pod \"maas-api-revoke-sub-85237e45-xbdk7\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:03.316312 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.316231 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xbdk7\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:03.316544 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.316341 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xbdk7\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:03.316544 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.316360 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2rs\" (UniqueName: \"kubernetes.io/projected/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-kube-api-access-7c2rs\") pod \"maas-api-revoke-sub-85237e45-xbdk7\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:03.316926 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.316889 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xbdk7\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:03.318683 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.318659 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xbdk7\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:03.324411 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.324393 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2rs\" (UniqueName: \"kubernetes.io/projected/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-kube-api-access-7c2rs\") pod \"maas-api-revoke-sub-85237e45-xbdk7\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:03.422286 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.422189 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:03.529519 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:02:03.529481 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:02:03.704878 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.704854 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7"] Nov 02 18:02:03.706865 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:02:03.706824 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9d464b_a820_45cc_a0ba_eb13fc6b9419.slice/crio-c15bc231c136d76e4234ea7d8cb23b5de52c06c51de0dc5c53d0f5cf09fcb4e6 WatchSource:0}: Error finding container c15bc231c136d76e4234ea7d8cb23b5de52c06c51de0dc5c53d0f5cf09fcb4e6: Status 404 returned error can't find the container with id c15bc231c136d76e4234ea7d8cb23b5de52c06c51de0dc5c53d0f5cf09fcb4e6 Nov 02 18:02:03.809428 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:03.809394 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" event={"ID":"6a9d464b-a820-45cc-a0ba-eb13fc6b9419","Type":"ContainerStarted","Data":"c15bc231c136d76e4234ea7d8cb23b5de52c06c51de0dc5c53d0f5cf09fcb4e6"} Nov 02 18:02:04.818278 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:04.818041 4310 generic.go:356] "Generic (PLEG): container finished" podID="6a9d464b-a820-45cc-a0ba-eb13fc6b9419" containerID="2e722dbc88531ef3020c2933ae2d5aa05b80b692ae1dccee3f058dc9681c4a72" exitCode=0 Nov 02 18:02:04.818278 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:04.818129 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" event={"ID":"6a9d464b-a820-45cc-a0ba-eb13fc6b9419","Type":"ContainerDied","Data":"2e722dbc88531ef3020c2933ae2d5aa05b80b692ae1dccee3f058dc9681c4a72"} Nov 02 18:02:04.960062 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:04.960030 4310 scope.go:122] "RemoveContainer" containerID="fb07e144101c63f16153c1311f099fa8e245830ff75606fb13892c88f0ff3695" Nov 02 18:02:04.971804 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:04.971786 4310 scope.go:122] "RemoveContainer" containerID="b5d97794db6cb57aa9d020a0167bf2b4d516688c9faddf1f3b320fe1b9638e63" Nov 02 18:02:04.981721 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:04.981699 4310 scope.go:122] "RemoveContainer" containerID="4cbdd708c7174dd94bdb2ca6edec9a8d0b5f0bf10b788d4543023421cd1a48b1" Nov 02 18:02:04.990172 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:04.990146 4310 scope.go:122] "RemoveContainer" containerID="204e3a6e88fc30f355337c50c815ce610ac3917010844d336ba606d38611cbce" Nov 02 18:02:04.998604 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:04.998583 4310 scope.go:122] "RemoveContainer" containerID="3d34b8dc514fd0089e09a79cb54494c24420fc573263a9853b99bba1bdfd05ca" Nov 02 18:02:05.007936 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:05.007907 4310 scope.go:122] "RemoveContainer" containerID="c4c6617d7e330dc9bdfde2d07b7ca48a3e0a26f97d0cb1e2d2337bf6cf4122aa" Nov 02 18:02:05.021685 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:05.021657 4310 scope.go:122] "RemoveContainer" containerID="9e2b8dd787b4e765fe3b9ae7437ca4e1424e48014563eddf8a8e26d7f4f77f36" Nov 02 18:02:05.030610 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:05.030589 4310 scope.go:122] "RemoveContainer" containerID="8717758ba259705e686bd00b13a9aed3317f1761b78d38ee38d92b905d9a4a15" Nov 02 18:02:05.966838 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:05.966813 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:06.024677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.024643 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq"] Nov 02 18:02:06.028842 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.028809 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fa5f65-f2psq"] Nov 02 18:02:06.042270 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.042239 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-xks-ca\") pod \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " Nov 02 18:02:06.042419 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.042340 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-kube-api-access-7c2rs\" (UniqueName: \"kubernetes.io/projected/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-kube-api-access-7c2rs\") pod \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " Nov 02 18:02:06.042419 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.042383 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-openshift-ca\") pod \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\" (UID: \"6a9d464b-a820-45cc-a0ba-eb13fc6b9419\") " Nov 02 18:02:06.042895 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.042863 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-openshift-ca" pod "6a9d464b-a820-45cc-a0ba-eb13fc6b9419" (UID: "6a9d464b-a820-45cc-a0ba-eb13fc6b9419"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:02:06.044405 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.044380 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-xks-ca" pod "6a9d464b-a820-45cc-a0ba-eb13fc6b9419" (UID: "6a9d464b-a820-45cc-a0ba-eb13fc6b9419"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:06.044534 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.044520 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-kube-api-access-7c2rs" pod "6a9d464b-a820-45cc-a0ba-eb13fc6b9419" (UID: "6a9d464b-a820-45cc-a0ba-eb13fc6b9419"). InnerVolumeSpecName "kube-api-access-7c2rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:06.143422 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.143342 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:06.143422 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.143374 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c2rs\" (UniqueName: \"kubernetes.io/projected/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-kube-api-access-7c2rs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:06.143422 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.143387 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a9d464b-a820-45cc-a0ba-eb13fc6b9419-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:06.672637 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.672601 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e328e69-eeb5-4258-81c3-7a56397c0563" path="/var/lib/kubelet/pods/6e328e69-eeb5-4258-81c3-7a56397c0563/volumes" Nov 02 18:02:06.831566 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.831539 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" Nov 02 18:02:06.831730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.831575 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7" event={"ID":"6a9d464b-a820-45cc-a0ba-eb13fc6b9419","Type":"ContainerDied","Data":"c15bc231c136d76e4234ea7d8cb23b5de52c06c51de0dc5c53d0f5cf09fcb4e6"} Nov 02 18:02:06.831730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:06.831626 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c15bc231c136d76e4234ea7d8cb23b5de52c06c51de0dc5c53d0f5cf09fcb4e6" Nov 02 18:02:14.680461 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:14.680429 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:14.812628 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:14.812585 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-698c564c8b-fvvw4"] Nov 02 18:02:14.812864 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:14.812831 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-698c564c8b-fvvw4" podUID="75e2fad2-7cea-4b95-ac23-3d98589b105b" containerName="manager" containerID="cri-o://0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93" gracePeriod=30 Nov 02 18:02:15.062785 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.062761 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:02:15.135337 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.135300 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-webhook-cert\") pod \"75e2fad2-7cea-4b95-ac23-3d98589b105b\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " Nov 02 18:02:15.135510 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.135350 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e2fad2-7cea-4b95-ac23-3d98589b105b-kube-api-access-rwsz6\" (UniqueName: \"kubernetes.io/projected/75e2fad2-7cea-4b95-ac23-3d98589b105b-kube-api-access-rwsz6\") pod \"75e2fad2-7cea-4b95-ac23-3d98589b105b\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " Nov 02 18:02:15.135510 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.135407 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-metrics-certs\") pod \"75e2fad2-7cea-4b95-ac23-3d98589b105b\" (UID: \"75e2fad2-7cea-4b95-ac23-3d98589b105b\") " Nov 02 18:02:15.137521 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.137489 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-webhook-cert" pod "75e2fad2-7cea-4b95-ac23-3d98589b105b" (UID: "75e2fad2-7cea-4b95-ac23-3d98589b105b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:15.137679 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.137571 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e2fad2-7cea-4b95-ac23-3d98589b105b-kube-api-access-rwsz6" pod "75e2fad2-7cea-4b95-ac23-3d98589b105b" (UID: "75e2fad2-7cea-4b95-ac23-3d98589b105b"). InnerVolumeSpecName "kube-api-access-rwsz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:15.137679 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.137584 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-metrics-certs" pod "75e2fad2-7cea-4b95-ac23-3d98589b105b" (UID: "75e2fad2-7cea-4b95-ac23-3d98589b105b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:15.236716 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.236676 4310 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-webhook-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:15.236716 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.236718 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwsz6\" (UniqueName: \"kubernetes.io/projected/75e2fad2-7cea-4b95-ac23-3d98589b105b-kube-api-access-rwsz6\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:15.236982 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.236737 4310 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e2fad2-7cea-4b95-ac23-3d98589b105b-metrics-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:02:15.880449 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.880411 4310 generic.go:356] "Generic (PLEG): container finished" podID="75e2fad2-7cea-4b95-ac23-3d98589b105b" containerID="0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93" exitCode=0 Nov 02 18:02:15.880895 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.880459 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-698c564c8b-fvvw4" Nov 02 18:02:15.880895 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.880501 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-698c564c8b-fvvw4" event={"ID":"75e2fad2-7cea-4b95-ac23-3d98589b105b","Type":"ContainerDied","Data":"0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93"} Nov 02 18:02:15.880895 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.880551 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-698c564c8b-fvvw4" event={"ID":"75e2fad2-7cea-4b95-ac23-3d98589b105b","Type":"ContainerDied","Data":"c81e78f09d7e5365425e3adc84b44e0f7855a9b3f0c7a6ce4471c610dc9b438a"} Nov 02 18:02:15.880895 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.880576 4310 scope.go:122] "RemoveContainer" containerID="0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93" Nov 02 18:02:15.890122 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.890101 4310 scope.go:122] "RemoveContainer" containerID="0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93" Nov 02 18:02:15.890423 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:02:15.890401 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93\": container with ID starting with 0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93 not found: ID does not exist" containerID="0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93" Nov 02 18:02:15.890493 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.890437 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93"} err="failed to get container status \"0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93\": rpc error: code = NotFound desc = could not find container \"0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93\": container with ID starting with 0aebd688994db5d8b2c074ecbba1f49b0bb23c0e3ffbed1b880a15c858abae93 not found: ID does not exist" Nov 02 18:02:15.904146 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.904095 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-698c564c8b-fvvw4"] Nov 02 18:02:15.906112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:15.906066 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-698c564c8b-fvvw4"] Nov 02 18:02:16.678175 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:16.673676 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e2fad2-7cea-4b95-ac23-3d98589b105b" path="/var/lib/kubelet/pods/75e2fad2-7cea-4b95-ac23-3d98589b105b/volumes" Nov 02 18:02:16.926425 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:16.926397 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:17.071959 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.071927 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-698c564c8b-rgrcd"] Nov 02 18:02:17.072363 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.072348 4310 cpu_manager.go:414] "containerMap: removing container" podUID="6a9d464b-a820-45cc-a0ba-eb13fc6b9419" containerName="revoke-keys" Nov 02 18:02:17.072427 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.072364 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9d464b-a820-45cc-a0ba-eb13fc6b9419" containerName="revoke-keys" Nov 02 18:02:17.072427 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.072385 4310 cpu_manager.go:414] "containerMap: removing container" podUID="75e2fad2-7cea-4b95-ac23-3d98589b105b" containerName="manager" Nov 02 18:02:17.072427 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.072390 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e2fad2-7cea-4b95-ac23-3d98589b105b" containerName="manager" Nov 02 18:02:17.072538 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.072460 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9d464b-a820-45cc-a0ba-eb13fc6b9419" containerName="revoke-keys" Nov 02 18:02:17.072538 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.072469 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e2fad2-7cea-4b95-ac23-3d98589b105b" containerName="manager" Nov 02 18:02:17.077048 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.077032 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:17.084520 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.084499 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 18:02:17.086387 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.086360 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 18:02:17.086552 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.086538 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5xmtd\"" Nov 02 18:02:17.086656 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.086551 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 18:02:17.088586 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.088565 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-698c564c8b-rgrcd"] Nov 02 18:02:17.153469 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.153433 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2qjr\" (UniqueName: \"kubernetes.io/projected/6058e7f5-08e4-47e6-a656-9fc94d877250-kube-api-access-z2qjr\") pod \"maas-controller-698c564c8b-rgrcd\" (UID: \"6058e7f5-08e4-47e6-a656-9fc94d877250\") " pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:17.153469 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.153471 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6058e7f5-08e4-47e6-a656-9fc94d877250-webhook-cert\") pod \"maas-controller-698c564c8b-rgrcd\" (UID: \"6058e7f5-08e4-47e6-a656-9fc94d877250\") " pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:17.153678 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.153586 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6058e7f5-08e4-47e6-a656-9fc94d877250-metrics-certs\") pod \"maas-controller-698c564c8b-rgrcd\" (UID: \"6058e7f5-08e4-47e6-a656-9fc94d877250\") " pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:17.254913 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.254858 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qjr\" (UniqueName: \"kubernetes.io/projected/6058e7f5-08e4-47e6-a656-9fc94d877250-kube-api-access-z2qjr\") pod \"maas-controller-698c564c8b-rgrcd\" (UID: \"6058e7f5-08e4-47e6-a656-9fc94d877250\") " pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:17.254913 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.254916 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6058e7f5-08e4-47e6-a656-9fc94d877250-webhook-cert\") pod \"maas-controller-698c564c8b-rgrcd\" (UID: \"6058e7f5-08e4-47e6-a656-9fc94d877250\") " pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:17.255168 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.254960 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6058e7f5-08e4-47e6-a656-9fc94d877250-metrics-certs\") pod \"maas-controller-698c564c8b-rgrcd\" (UID: \"6058e7f5-08e4-47e6-a656-9fc94d877250\") " pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:17.257430 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.257407 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6058e7f5-08e4-47e6-a656-9fc94d877250-metrics-certs\") pod \"maas-controller-698c564c8b-rgrcd\" (UID: \"6058e7f5-08e4-47e6-a656-9fc94d877250\") " pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:17.257546 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.257458 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6058e7f5-08e4-47e6-a656-9fc94d877250-webhook-cert\") pod \"maas-controller-698c564c8b-rgrcd\" (UID: \"6058e7f5-08e4-47e6-a656-9fc94d877250\") " pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:17.265054 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.265029 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qjr\" (UniqueName: \"kubernetes.io/projected/6058e7f5-08e4-47e6-a656-9fc94d877250-kube-api-access-z2qjr\") pod \"maas-controller-698c564c8b-rgrcd\" (UID: \"6058e7f5-08e4-47e6-a656-9fc94d877250\") " pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:17.391203 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.391115 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:17.668896 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.668873 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-698c564c8b-rgrcd"] Nov 02 18:02:17.670938 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:02:17.670903 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6058e7f5_08e4_47e6_a656_9fc94d877250.slice/crio-23652f1ef8efb52821ece4405697c95e871d42c528d94125d271f9f67ece27c0 WatchSource:0}: Error finding container 23652f1ef8efb52821ece4405697c95e871d42c528d94125d271f9f67ece27c0: Status 404 returned error can't find the container with id 23652f1ef8efb52821ece4405697c95e871d42c528d94125d271f9f67ece27c0 Nov 02 18:02:17.893645 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:17.893611 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-698c564c8b-rgrcd" event={"ID":"6058e7f5-08e4-47e6-a656-9fc94d877250","Type":"ContainerStarted","Data":"23652f1ef8efb52821ece4405697c95e871d42c528d94125d271f9f67ece27c0"} Nov 02 18:02:18.901203 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:18.901167 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-698c564c8b-rgrcd" event={"ID":"6058e7f5-08e4-47e6-a656-9fc94d877250","Type":"ContainerStarted","Data":"de9e0a6d18697ba8586b3a4192cde44bb777d89638751ed4bbf6f11a2ca09392"} Nov 02 18:02:18.901594 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:18.901345 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:18.920860 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:18.920809 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-698c564c8b-rgrcd" podStartSLOduration=1.491479658 podStartE2EDuration="1.920796181s" totalImagesPullingTime="429.316523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:18.918457993 +0000 UTC m=+1336.361540629" watchObservedRunningTime="2026-11-02 18:02:18.920796181 +0000 UTC m=+1336.363878818" Nov 02 18:02:24.026705 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:24.026656 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:29.908223 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:29.908188 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-698c564c8b-rgrcd" Nov 02 18:02:33.522965 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:02:33.522931 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:02:43.037394 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:43.037347 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn"] Nov 02 18:02:43.043245 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:43.043220 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-924dc11d-shxxn"] Nov 02 18:02:44.673155 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:44.673115 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd331b6-2583-4ea1-a326-3d18f31321c0" path="/var/lib/kubelet/pods/7fd331b6-2583-4ea1-a326-3d18f31321c0/volumes" Nov 02 18:02:48.199603 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:48.199567 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:02:56.030832 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:56.030797 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk"] Nov 02 18:02:56.032689 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:56.032664 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm"] Nov 02 18:02:56.036477 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:56.036451 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf201658-v6vrk"] Nov 02 18:02:56.038058 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:56.038040 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e41d10-cwsmm"] Nov 02 18:02:56.673729 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:56.673696 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ef9be3-7c5b-4407-b20b-277501637575" path="/var/lib/kubelet/pods/82ef9be3-7c5b-4407-b20b-277501637575/volumes" Nov 02 18:02:56.674133 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:56.674119 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f79edd-d030-40b3-a593-c6ea421cef43" path="/var/lib/kubelet/pods/d9f79edd-d030-40b3-a593-c6ea421cef43/volumes" Nov 02 18:02:57.029394 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.029359 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg"] Nov 02 18:02:57.033044 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.033026 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:02:57.041460 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.041431 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:02:57.042715 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.042641 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:02:57.047036 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.047009 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg"] Nov 02 18:02:57.072934 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.072880 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xtdsg\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:02:57.073139 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.073018 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xtdsg\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:02:57.073220 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.073145 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsc4x\" (UniqueName: \"kubernetes.io/projected/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-kube-api-access-zsc4x\") pod \"maas-api-revoke-sub-c1e271a9-xtdsg\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:02:57.173851 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.173811 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xtdsg\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:02:57.174029 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.173884 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xtdsg\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:02:57.174029 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.173933 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsc4x\" (UniqueName: \"kubernetes.io/projected/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-kube-api-access-zsc4x\") pod \"maas-api-revoke-sub-c1e271a9-xtdsg\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:02:57.174484 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.174461 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xtdsg\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:02:57.176272 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.176249 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xtdsg\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:02:57.183931 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.183911 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsc4x\" (UniqueName: \"kubernetes.io/projected/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-kube-api-access-zsc4x\") pod \"maas-api-revoke-sub-c1e271a9-xtdsg\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:02:57.358213 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.358127 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:02:57.642028 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:57.642003 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg"] Nov 02 18:02:57.644169 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:02:57.644140 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d51cd7f_a7a5_4803_b6c6_981fb0831b12.slice/crio-7e141b61287736895f5dbf84527162df5eeddd242cba548fe5e3cea750071752 WatchSource:0}: Error finding container 7e141b61287736895f5dbf84527162df5eeddd242cba548fe5e3cea750071752: Status 404 returned error can't find the container with id 7e141b61287736895f5dbf84527162df5eeddd242cba548fe5e3cea750071752 Nov 02 18:02:58.150367 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:58.150325 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" event={"ID":"5d51cd7f-a7a5-4803-b6c6-981fb0831b12","Type":"ContainerStarted","Data":"6c166379e02691394f717c672fd630bb3c0aa4a97cce75d34ad76c1b1c527dbb"} Nov 02 18:02:58.150829 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:58.150386 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" event={"ID":"5d51cd7f-a7a5-4803-b6c6-981fb0831b12","Type":"ContainerStarted","Data":"7e141b61287736895f5dbf84527162df5eeddd242cba548fe5e3cea750071752"} Nov 02 18:02:58.173806 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:58.173754 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" podStartSLOduration=1.173740057 podStartE2EDuration="1.173740057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:58.172490521 +0000 UTC m=+1375.615573159" watchObservedRunningTime="2026-11-02 18:02:58.173740057 +0000 UTC m=+1375.616822753" Nov 02 18:02:59.157602 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:59.157563 4310 generic.go:356] "Generic (PLEG): container finished" podID="5d51cd7f-a7a5-4803-b6c6-981fb0831b12" containerID="6c166379e02691394f717c672fd630bb3c0aa4a97cce75d34ad76c1b1c527dbb" exitCode=0 Nov 02 18:02:59.158020 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:02:59.157650 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" event={"ID":"5d51cd7f-a7a5-4803-b6c6-981fb0831b12","Type":"ContainerDied","Data":"6c166379e02691394f717c672fd630bb3c0aa4a97cce75d34ad76c1b1c527dbb"} Nov 02 18:03:00.302353 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:00.302325 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:03:00.398861 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:00.398817 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-xks-ca\") pod \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " Nov 02 18:03:00.399101 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:00.398869 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-kube-api-access-zsc4x\" (UniqueName: \"kubernetes.io/projected/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-kube-api-access-zsc4x\") pod \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " Nov 02 18:03:00.399101 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:00.398941 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-openshift-ca\") pod \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\" (UID: \"5d51cd7f-a7a5-4803-b6c6-981fb0831b12\") " Nov 02 18:03:00.399374 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:00.399351 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-openshift-ca" pod "5d51cd7f-a7a5-4803-b6c6-981fb0831b12" (UID: "5d51cd7f-a7a5-4803-b6c6-981fb0831b12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:00.401048 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:00.401021 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-xks-ca" pod "5d51cd7f-a7a5-4803-b6c6-981fb0831b12" (UID: "5d51cd7f-a7a5-4803-b6c6-981fb0831b12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:03:00.401145 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:00.401096 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-kube-api-access-zsc4x" pod "5d51cd7f-a7a5-4803-b6c6-981fb0831b12" (UID: "5d51cd7f-a7a5-4803-b6c6-981fb0831b12"). InnerVolumeSpecName "kube-api-access-zsc4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:00.500221 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:00.500183 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:00.500221 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:00.500213 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsc4x\" (UniqueName: \"kubernetes.io/projected/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-kube-api-access-zsc4x\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:00.500221 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:00.500225 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d51cd7f-a7a5-4803-b6c6-981fb0831b12-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:01.170676 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:01.170641 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" Nov 02 18:03:01.170905 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:01.170674 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg" event={"ID":"5d51cd7f-a7a5-4803-b6c6-981fb0831b12","Type":"ContainerDied","Data":"7e141b61287736895f5dbf84527162df5eeddd242cba548fe5e3cea750071752"} Nov 02 18:03:01.170905 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:01.170723 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e141b61287736895f5dbf84527162df5eeddd242cba548fe5e3cea750071752" Nov 02 18:03:03.523054 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:03:03.523014 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:03:05.108823 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:05.108791 4310 scope.go:122] "RemoveContainer" containerID="595014f621ca1b210bf38da9c0d5766efd8251f8888fdfbbf0dfa69c45b22536" Nov 02 18:03:05.117892 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:05.117872 4310 scope.go:122] "RemoveContainer" containerID="0cc318c9075f213f9eef1f2d5d6282ec3cc6168637b95fb443f7c6c03d3eb9b2" Nov 02 18:03:05.125857 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:05.125841 4310 scope.go:122] "RemoveContainer" containerID="0a0b3918d8b6ad14c50d7ddc205dfe8be91441394c05b17e027c2e86223f8078" Nov 02 18:03:05.134030 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:05.134006 4310 scope.go:122] "RemoveContainer" containerID="1b52e01cda912fad94a201d6c8968cdfcf702c6431f58f95d96622a0e116c460" Nov 02 18:03:05.141607 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:05.141590 4310 scope.go:122] "RemoveContainer" containerID="0cfe717ef10377f9eaae036686b70b5e2790b87b591ec2d44b5078d144916f23" Nov 02 18:03:05.148977 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:05.148956 4310 scope.go:122] "RemoveContainer" containerID="1b6d3cc6bba1ce9445fc2272044c70e5db0835bfb38285f46010e35df003f26a" Nov 02 18:03:05.156619 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:05.156602 4310 scope.go:122] "RemoveContainer" containerID="cf822c6e1272407d38e23ae4aaca99a256f14ab0f3d50ad785c947ec6ddd8d10" Nov 02 18:03:05.164556 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:05.164531 4310 scope.go:122] "RemoveContainer" containerID="04f3ab1bb73daf630578e517a7ad1e09dd6979f3a58ab043d2e45edc48c8960c" Nov 02 18:03:05.172521 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:05.172502 4310 scope.go:122] "RemoveContainer" containerID="509af34add89ddc522002157c0349cd3cc11da75e4c38561d7a56edab9861328" Nov 02 18:03:14.157112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:14.156199 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:03:14.157112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:14.156279 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:17.974573 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:17.974484 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj"] Nov 02 18:03:17.975193 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:17.975168 4310 cpu_manager.go:414] "containerMap: removing container" podUID="5d51cd7f-a7a5-4803-b6c6-981fb0831b12" containerName="revoke-keys" Nov 02 18:03:17.975246 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:17.975198 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d51cd7f-a7a5-4803-b6c6-981fb0831b12" containerName="revoke-keys" Nov 02 18:03:17.975348 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:17.975336 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d51cd7f-a7a5-4803-b6c6-981fb0831b12" containerName="revoke-keys" Nov 02 18:03:17.981009 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:17.980990 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:17.993157 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:17.993104 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:03:17.993443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:17.993419 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj"] Nov 02 18:03:17.995465 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:17.995445 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:03:18.066396 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:18.066361 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lggcj\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:18.066396 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:18.066397 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lggcj\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:18.066609 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:18.066512 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngwmd\" (UniqueName: \"kubernetes.io/projected/a5237c66-6b6e-4279-88c4-e826ed10d598-kube-api-access-ngwmd\") pod \"maas-api-revoke-sub-2e1f5b5f-lggcj\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:18.167337 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:18.167303 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngwmd\" (UniqueName: \"kubernetes.io/projected/a5237c66-6b6e-4279-88c4-e826ed10d598-kube-api-access-ngwmd\") pod \"maas-api-revoke-sub-2e1f5b5f-lggcj\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:18.167550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:18.167423 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lggcj\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:18.167550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:18.167454 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lggcj\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:18.168190 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:18.168161 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lggcj\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:18.169735 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:18.169703 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lggcj\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:18.179412 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:18.179388 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngwmd\" (UniqueName: \"kubernetes.io/projected/a5237c66-6b6e-4279-88c4-e826ed10d598-kube-api-access-ngwmd\") pod \"maas-api-revoke-sub-2e1f5b5f-lggcj\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:18.294448 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:18.294365 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:18.574073 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:18.573990 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj"] Nov 02 18:03:18.577160 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:03:18.577130 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5237c66_6b6e_4279_88c4_e826ed10d598.slice/crio-27214b951783f63a2888e77f3c1f1e2b7a9e12a0048722fff7d3f971fe67f913 WatchSource:0}: Error finding container 27214b951783f63a2888e77f3c1f1e2b7a9e12a0048722fff7d3f971fe67f913: Status 404 returned error can't find the container with id 27214b951783f63a2888e77f3c1f1e2b7a9e12a0048722fff7d3f971fe67f913 Nov 02 18:03:19.286358 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:19.286319 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" event={"ID":"a5237c66-6b6e-4279-88c4-e826ed10d598","Type":"ContainerStarted","Data":"7d2620c02ebabec1dfe67e63102b342b6ea27a39ff8f7c06528cf319abbe4eff"} Nov 02 18:03:19.286751 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:19.286387 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" event={"ID":"a5237c66-6b6e-4279-88c4-e826ed10d598","Type":"ContainerStarted","Data":"27214b951783f63a2888e77f3c1f1e2b7a9e12a0048722fff7d3f971fe67f913"} Nov 02 18:03:19.307747 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:19.307699 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" podStartSLOduration=2.307685608 podStartE2EDuration="2.307685608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:19.30566413 +0000 UTC m=+1396.748746766" watchObservedRunningTime="2026-11-02 18:03:19.307685608 +0000 UTC m=+1396.750768271" Nov 02 18:03:20.299503 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:20.299465 4310 generic.go:356] "Generic (PLEG): container finished" podID="a5237c66-6b6e-4279-88c4-e826ed10d598" containerID="7d2620c02ebabec1dfe67e63102b342b6ea27a39ff8f7c06528cf319abbe4eff" exitCode=0 Nov 02 18:03:20.299929 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:20.299540 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" event={"ID":"a5237c66-6b6e-4279-88c4-e826ed10d598","Type":"ContainerDied","Data":"7d2620c02ebabec1dfe67e63102b342b6ea27a39ff8f7c06528cf319abbe4eff"} Nov 02 18:03:21.443488 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:21.443464 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:21.502115 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:21.502072 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-xks-ca\") pod \"a5237c66-6b6e-4279-88c4-e826ed10d598\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " Nov 02 18:03:21.502289 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:21.502152 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-openshift-ca\") pod \"a5237c66-6b6e-4279-88c4-e826ed10d598\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " Nov 02 18:03:21.502355 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:21.502338 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5237c66-6b6e-4279-88c4-e826ed10d598-kube-api-access-ngwmd\" (UniqueName: \"kubernetes.io/projected/a5237c66-6b6e-4279-88c4-e826ed10d598-kube-api-access-ngwmd\") pod \"a5237c66-6b6e-4279-88c4-e826ed10d598\" (UID: \"a5237c66-6b6e-4279-88c4-e826ed10d598\") " Nov 02 18:03:21.502512 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:21.502492 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-openshift-ca" pod "a5237c66-6b6e-4279-88c4-e826ed10d598" (UID: "a5237c66-6b6e-4279-88c4-e826ed10d598"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:21.502758 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:21.502736 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:21.504258 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:21.504233 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-xks-ca" pod "a5237c66-6b6e-4279-88c4-e826ed10d598" (UID: "a5237c66-6b6e-4279-88c4-e826ed10d598"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:03:21.504385 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:21.504355 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5237c66-6b6e-4279-88c4-e826ed10d598-kube-api-access-ngwmd" pod "a5237c66-6b6e-4279-88c4-e826ed10d598" (UID: "a5237c66-6b6e-4279-88c4-e826ed10d598"). InnerVolumeSpecName "kube-api-access-ngwmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:21.604293 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:21.604204 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngwmd\" (UniqueName: \"kubernetes.io/projected/a5237c66-6b6e-4279-88c4-e826ed10d598-kube-api-access-ngwmd\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:21.604293 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:21.604239 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5237c66-6b6e-4279-88c4-e826ed10d598-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:22.312655 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:22.312618 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" event={"ID":"a5237c66-6b6e-4279-88c4-e826ed10d598","Type":"ContainerDied","Data":"27214b951783f63a2888e77f3c1f1e2b7a9e12a0048722fff7d3f971fe67f913"} Nov 02 18:03:22.312655 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:22.312640 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj" Nov 02 18:03:22.312871 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:22.312672 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27214b951783f63a2888e77f3c1f1e2b7a9e12a0048722fff7d3f971fe67f913" Nov 02 18:03:28.962739 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:28.962700 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn"] Nov 02 18:03:28.963156 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:28.963116 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a5237c66-6b6e-4279-88c4-e826ed10d598" containerName="revoke-keys" Nov 02 18:03:28.963156 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:28.963129 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5237c66-6b6e-4279-88c4-e826ed10d598" containerName="revoke-keys" Nov 02 18:03:28.963292 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:28.963246 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5237c66-6b6e-4279-88c4-e826ed10d598" containerName="revoke-keys" Nov 02 18:03:28.966513 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:28.966493 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:28.971740 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:28.971714 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:03:28.972568 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:28.972545 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:03:28.977912 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:28.977884 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn"] Nov 02 18:03:29.073363 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:29.073309 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4f6h\" (UniqueName: \"kubernetes.io/projected/9a112456-8204-4567-871e-5be0fe6a7e7e-kube-api-access-j4f6h\") pod \"maas-api-revoke-sub-ec843018-zj4vn\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:29.073584 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:29.073500 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zj4vn\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:29.073665 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:29.073647 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zj4vn\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:29.175210 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:29.175175 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zj4vn\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:29.175398 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:29.175257 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zj4vn\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:29.175398 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:29.175278 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4f6h\" (UniqueName: \"kubernetes.io/projected/9a112456-8204-4567-871e-5be0fe6a7e7e-kube-api-access-j4f6h\") pod \"maas-api-revoke-sub-ec843018-zj4vn\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:29.175840 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:29.175806 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zj4vn\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:29.177526 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:29.177504 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zj4vn\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:29.187258 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:29.187233 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4f6h\" (UniqueName: \"kubernetes.io/projected/9a112456-8204-4567-871e-5be0fe6a7e7e-kube-api-access-j4f6h\") pod \"maas-api-revoke-sub-ec843018-zj4vn\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:29.282847 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:29.282747 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:29.579838 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:29.579807 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn"] Nov 02 18:03:29.581672 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:03:29.581649 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a112456_8204_4567_871e_5be0fe6a7e7e.slice/crio-868890f3ce10345854f011a706c8d16890c89500f400e3755c1d10eea38fa1f4 WatchSource:0}: Error finding container 868890f3ce10345854f011a706c8d16890c89500f400e3755c1d10eea38fa1f4: Status 404 returned error can't find the container with id 868890f3ce10345854f011a706c8d16890c89500f400e3755c1d10eea38fa1f4 Nov 02 18:03:30.355177 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:30.355140 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" event={"ID":"9a112456-8204-4567-871e-5be0fe6a7e7e","Type":"ContainerStarted","Data":"fcad6a36ec9fb3b681e7acc7885f51d136a2fa9676fd4ba8b419f5cc6470b69a"} Nov 02 18:03:30.355649 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:30.355190 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" event={"ID":"9a112456-8204-4567-871e-5be0fe6a7e7e","Type":"ContainerStarted","Data":"868890f3ce10345854f011a706c8d16890c89500f400e3755c1d10eea38fa1f4"} Nov 02 18:03:30.376603 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:30.376552 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" podStartSLOduration=2.376538579 podStartE2EDuration="2.376538579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:30.376339533 +0000 UTC m=+1407.819422170" watchObservedRunningTime="2026-11-02 18:03:30.376538579 +0000 UTC m=+1407.819621212" Nov 02 18:03:31.364737 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:31.364698 4310 generic.go:356] "Generic (PLEG): container finished" podID="9a112456-8204-4567-871e-5be0fe6a7e7e" containerID="fcad6a36ec9fb3b681e7acc7885f51d136a2fa9676fd4ba8b419f5cc6470b69a" exitCode=0 Nov 02 18:03:31.365213 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:31.364792 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" event={"ID":"9a112456-8204-4567-871e-5be0fe6a7e7e","Type":"ContainerDied","Data":"fcad6a36ec9fb3b681e7acc7885f51d136a2fa9676fd4ba8b419f5cc6470b69a"} Nov 02 18:03:32.824812 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:32.824775 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:32.833564 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:32.833536 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a112456-8204-4567-871e-5be0fe6a7e7e-kube-api-access-j4f6h\" (UniqueName: \"kubernetes.io/projected/9a112456-8204-4567-871e-5be0fe6a7e7e-kube-api-access-j4f6h\") pod \"9a112456-8204-4567-871e-5be0fe6a7e7e\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " Nov 02 18:03:32.833707 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:32.833679 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-openshift-ca\") pod \"9a112456-8204-4567-871e-5be0fe6a7e7e\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " Nov 02 18:03:32.833779 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:32.833721 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-xks-ca\") pod \"9a112456-8204-4567-871e-5be0fe6a7e7e\" (UID: \"9a112456-8204-4567-871e-5be0fe6a7e7e\") " Nov 02 18:03:32.834510 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:32.834484 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-openshift-ca" pod "9a112456-8204-4567-871e-5be0fe6a7e7e" (UID: "9a112456-8204-4567-871e-5be0fe6a7e7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:32.835972 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:32.835947 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a112456-8204-4567-871e-5be0fe6a7e7e-kube-api-access-j4f6h" pod "9a112456-8204-4567-871e-5be0fe6a7e7e" (UID: "9a112456-8204-4567-871e-5be0fe6a7e7e"). InnerVolumeSpecName "kube-api-access-j4f6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:32.836067 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:32.835980 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-xks-ca" pod "9a112456-8204-4567-871e-5be0fe6a7e7e" (UID: "9a112456-8204-4567-871e-5be0fe6a7e7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:03:32.934895 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:32.934861 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:32.934895 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:32.934892 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a112456-8204-4567-871e-5be0fe6a7e7e-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:32.934895 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:32.934902 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4f6h\" (UniqueName: \"kubernetes.io/projected/9a112456-8204-4567-871e-5be0fe6a7e7e-kube-api-access-j4f6h\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:33.420053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:33.420011 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" event={"ID":"9a112456-8204-4567-871e-5be0fe6a7e7e","Type":"ContainerDied","Data":"868890f3ce10345854f011a706c8d16890c89500f400e3755c1d10eea38fa1f4"} Nov 02 18:03:33.421424 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:33.421258 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="868890f3ce10345854f011a706c8d16890c89500f400e3755c1d10eea38fa1f4" Nov 02 18:03:33.421424 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:33.421191 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn" Nov 02 18:03:33.538193 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:03:33.538159 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:03:35.479428 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:35.479396 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:36.452455 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:36.452422 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:44.156102 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:44.155242 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:03:44.156102 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:44.155326 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:44.763122 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:44.762198 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8ftbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:03:44.763122 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:44.762271 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8ftbd" podUID="8d9f41e4-d435-4f1f-95dc-858820a331bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:03:44.773298 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:44.770274 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:48.208913 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:48.208873 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:03:55.184865 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.184824 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv"] Nov 02 18:03:55.185459 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.185438 4310 cpu_manager.go:414] "containerMap: removing container" podUID="9a112456-8204-4567-871e-5be0fe6a7e7e" containerName="revoke-keys" Nov 02 18:03:55.185542 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.185462 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a112456-8204-4567-871e-5be0fe6a7e7e" containerName="revoke-keys" Nov 02 18:03:55.185602 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.185583 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a112456-8204-4567-871e-5be0fe6a7e7e" containerName="revoke-keys" Nov 02 18:03:55.188628 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.188606 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:55.196477 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.196454 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:03:55.197922 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.197901 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:03:55.201026 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.201004 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv"] Nov 02 18:03:55.274692 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.274657 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-746lv\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:55.274897 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.274744 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-746lv\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:55.274897 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.274831 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbm9n\" (UniqueName: \"kubernetes.io/projected/d7034824-3dff-401e-8d5b-89816b1040da-kube-api-access-lbm9n\") pod \"maas-api-revoke-sub-046e1343-746lv\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:55.376340 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.376308 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbm9n\" (UniqueName: \"kubernetes.io/projected/d7034824-3dff-401e-8d5b-89816b1040da-kube-api-access-lbm9n\") pod \"maas-api-revoke-sub-046e1343-746lv\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:55.376551 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.376390 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-746lv\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:55.376551 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.376452 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-746lv\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:55.377103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.377060 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-746lv\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:55.378759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.378737 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-746lv\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:55.385019 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.384997 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbm9n\" (UniqueName: \"kubernetes.io/projected/d7034824-3dff-401e-8d5b-89816b1040da-kube-api-access-lbm9n\") pod \"maas-api-revoke-sub-046e1343-746lv\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:55.503187 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.503152 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:55.782247 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:55.782173 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv"] Nov 02 18:03:55.790091 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:03:55.790045 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7034824_3dff_401e_8d5b_89816b1040da.slice/crio-50faf3edc83372a2cc1470c410ee66bb01cd63137f5fb0499f63d87567e77891 WatchSource:0}: Error finding container 50faf3edc83372a2cc1470c410ee66bb01cd63137f5fb0499f63d87567e77891: Status 404 returned error can't find the container with id 50faf3edc83372a2cc1470c410ee66bb01cd63137f5fb0499f63d87567e77891 Nov 02 18:03:56.581063 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:56.581033 4310 generic.go:356] "Generic (PLEG): container finished" podID="d7034824-3dff-401e-8d5b-89816b1040da" containerID="df0def6848b4461775c2f1b18ba44e26e31e32b90d177f12653ed86ab36cdd8c" exitCode=0 Nov 02 18:03:56.581439 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:56.581112 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" event={"ID":"d7034824-3dff-401e-8d5b-89816b1040da","Type":"ContainerDied","Data":"df0def6848b4461775c2f1b18ba44e26e31e32b90d177f12653ed86ab36cdd8c"} Nov 02 18:03:56.581439 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:56.581196 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" event={"ID":"d7034824-3dff-401e-8d5b-89816b1040da","Type":"ContainerStarted","Data":"50faf3edc83372a2cc1470c410ee66bb01cd63137f5fb0499f63d87567e77891"} Nov 02 18:03:57.727204 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:57.727172 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:57.900233 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:57.900145 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7034824-3dff-401e-8d5b-89816b1040da-kube-api-access-lbm9n\" (UniqueName: \"kubernetes.io/projected/d7034824-3dff-401e-8d5b-89816b1040da-kube-api-access-lbm9n\") pod \"d7034824-3dff-401e-8d5b-89816b1040da\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " Nov 02 18:03:57.900398 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:57.900335 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-openshift-ca\") pod \"d7034824-3dff-401e-8d5b-89816b1040da\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " Nov 02 18:03:57.900398 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:57.900392 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-xks-ca\") pod \"d7034824-3dff-401e-8d5b-89816b1040da\" (UID: \"d7034824-3dff-401e-8d5b-89816b1040da\") " Nov 02 18:03:57.900711 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:57.900685 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-openshift-ca" pod "d7034824-3dff-401e-8d5b-89816b1040da" (UID: "d7034824-3dff-401e-8d5b-89816b1040da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:57.902268 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:57.902247 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7034824-3dff-401e-8d5b-89816b1040da-kube-api-access-lbm9n" pod "d7034824-3dff-401e-8d5b-89816b1040da" (UID: "d7034824-3dff-401e-8d5b-89816b1040da"). InnerVolumeSpecName "kube-api-access-lbm9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:57.902342 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:57.902316 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-xks-ca" pod "d7034824-3dff-401e-8d5b-89816b1040da" (UID: "d7034824-3dff-401e-8d5b-89816b1040da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:03:58.001832 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:58.001790 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbm9n\" (UniqueName: \"kubernetes.io/projected/d7034824-3dff-401e-8d5b-89816b1040da-kube-api-access-lbm9n\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:58.001832 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:58.001824 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:58.001832 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:58.001837 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7034824-3dff-401e-8d5b-89816b1040da-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:03:58.592509 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:58.592477 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" Nov 02 18:03:58.592509 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:58.592504 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv" event={"ID":"d7034824-3dff-401e-8d5b-89816b1040da","Type":"ContainerDied","Data":"50faf3edc83372a2cc1470c410ee66bb01cd63137f5fb0499f63d87567e77891"} Nov 02 18:03:58.592734 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:03:58.592555 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50faf3edc83372a2cc1470c410ee66bb01cd63137f5fb0499f63d87567e77891" Nov 02 18:04:03.522027 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:04:03.521988 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:04:05.231183 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.231148 4310 scope.go:122] "RemoveContainer" containerID="0b64e8b19a55878b3f49b04b3e07f162207f89e79b368708489d4180552c461f" Nov 02 18:04:05.239949 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.239926 4310 scope.go:122] "RemoveContainer" containerID="e7f86aa3d92cf05cbd7c7be69730c59814a2f396ba2777d917a5228608e2f384" Nov 02 18:04:05.247926 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.247908 4310 scope.go:122] "RemoveContainer" containerID="41b033bf84ef4d24c01ede05e3dfd0162a345df3df613307eca344b44e44a2d8" Nov 02 18:04:05.255750 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.255722 4310 scope.go:122] "RemoveContainer" containerID="2a26ae200905a2db98ac83a84eb9e540d15fb8c78b29c587d0095e4e21c6b77a" Nov 02 18:04:05.266589 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.266568 4310 scope.go:122] "RemoveContainer" containerID="3e4f6488223ab9f201f0d96d0993d108d9ac2a574ad818ee4b786c3a26f438a2" Nov 02 18:04:05.508165 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.508067 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds"] Nov 02 18:04:05.508543 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.508529 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d7034824-3dff-401e-8d5b-89816b1040da" containerName="revoke-keys" Nov 02 18:04:05.508597 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.508545 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7034824-3dff-401e-8d5b-89816b1040da" containerName="revoke-keys" Nov 02 18:04:05.508633 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.508600 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7034824-3dff-401e-8d5b-89816b1040da" containerName="revoke-keys" Nov 02 18:04:05.513512 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.513489 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:05.520507 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.520476 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:04:05.521584 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.520702 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:04:05.523232 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.523206 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds"] Nov 02 18:04:05.574670 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.574633 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqmf8\" (UniqueName: \"kubernetes.io/projected/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-kube-api-access-hqmf8\") pod \"maas-api-revoke-sub-d3fba056-wlkds\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:05.574851 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.574680 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-wlkds\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:05.574851 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.574697 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-wlkds\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:05.675550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.675516 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqmf8\" (UniqueName: \"kubernetes.io/projected/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-kube-api-access-hqmf8\") pod \"maas-api-revoke-sub-d3fba056-wlkds\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:05.675723 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.675564 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-wlkds\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:05.675723 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.675593 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-wlkds\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:05.676247 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.676217 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-wlkds\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:05.677954 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.677935 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-wlkds\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:05.685890 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.685860 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqmf8\" (UniqueName: \"kubernetes.io/projected/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-kube-api-access-hqmf8\") pod \"maas-api-revoke-sub-d3fba056-wlkds\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:05.829747 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:05.829645 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:06.109601 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:06.109575 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds"] Nov 02 18:04:06.111155 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:04:06.111123 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c8f542_1636_424a_bfcf_8d4aa86e3cdb.slice/crio-69c33650aeeb48c2e322a54b92e8a433ca08fe0db484fa36c12fd2264fc14247 WatchSource:0}: Error finding container 69c33650aeeb48c2e322a54b92e8a433ca08fe0db484fa36c12fd2264fc14247: Status 404 returned error can't find the container with id 69c33650aeeb48c2e322a54b92e8a433ca08fe0db484fa36c12fd2264fc14247 Nov 02 18:04:06.635823 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:06.635719 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" event={"ID":"59c8f542-1636-424a-bfcf-8d4aa86e3cdb","Type":"ContainerStarted","Data":"f9c6acfc36cb97d4fd2538fd76bce9796df45cf7c0e711157dba9788c584b606"} Nov 02 18:04:06.635823 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:06.635800 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" event={"ID":"59c8f542-1636-424a-bfcf-8d4aa86e3cdb","Type":"ContainerStarted","Data":"69c33650aeeb48c2e322a54b92e8a433ca08fe0db484fa36c12fd2264fc14247"} Nov 02 18:04:06.657238 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:06.657182 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" podStartSLOduration=1.6571689410000001 podStartE2EDuration="1.657168941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:06.653892207 +0000 UTC m=+1444.096974844" watchObservedRunningTime="2026-11-02 18:04:06.657168941 +0000 UTC m=+1444.100251596" Nov 02 18:04:07.642961 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:07.642410 4310 generic.go:356] "Generic (PLEG): container finished" podID="59c8f542-1636-424a-bfcf-8d4aa86e3cdb" containerID="f9c6acfc36cb97d4fd2538fd76bce9796df45cf7c0e711157dba9788c584b606" exitCode=0 Nov 02 18:04:07.642961 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:07.642500 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" event={"ID":"59c8f542-1636-424a-bfcf-8d4aa86e3cdb","Type":"ContainerDied","Data":"f9c6acfc36cb97d4fd2538fd76bce9796df45cf7c0e711157dba9788c584b606"} Nov 02 18:04:08.976266 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:08.976237 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:09.096443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.096405 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-openshift-ca\") pod \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " Nov 02 18:04:09.096644 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.096478 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-xks-ca\") pod \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " Nov 02 18:04:09.096644 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.096550 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-kube-api-access-hqmf8\" (UniqueName: \"kubernetes.io/projected/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-kube-api-access-hqmf8\") pod \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\" (UID: \"59c8f542-1636-424a-bfcf-8d4aa86e3cdb\") " Nov 02 18:04:09.096803 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.096777 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-openshift-ca" pod "59c8f542-1636-424a-bfcf-8d4aa86e3cdb" (UID: "59c8f542-1636-424a-bfcf-8d4aa86e3cdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:09.096941 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.096928 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:04:09.098655 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.098632 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-kube-api-access-hqmf8" pod "59c8f542-1636-424a-bfcf-8d4aa86e3cdb" (UID: "59c8f542-1636-424a-bfcf-8d4aa86e3cdb"). InnerVolumeSpecName "kube-api-access-hqmf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:09.098655 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.098646 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-xks-ca" pod "59c8f542-1636-424a-bfcf-8d4aa86e3cdb" (UID: "59c8f542-1636-424a-bfcf-8d4aa86e3cdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:09.198419 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.198378 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqmf8\" (UniqueName: \"kubernetes.io/projected/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-kube-api-access-hqmf8\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:04:09.198419 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.198412 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c8f542-1636-424a-bfcf-8d4aa86e3cdb-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:04:09.660802 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.660767 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" Nov 02 18:04:09.660995 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.660803 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds" event={"ID":"59c8f542-1636-424a-bfcf-8d4aa86e3cdb","Type":"ContainerDied","Data":"69c33650aeeb48c2e322a54b92e8a433ca08fe0db484fa36c12fd2264fc14247"} Nov 02 18:04:09.660995 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:09.660855 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69c33650aeeb48c2e322a54b92e8a433ca08fe0db484fa36c12fd2264fc14247" Nov 02 18:04:33.523883 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:04:33.523844 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:04:36.421136 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.421042 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff"] Nov 02 18:04:36.421650 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.421499 4310 cpu_manager.go:414] "containerMap: removing container" podUID="59c8f542-1636-424a-bfcf-8d4aa86e3cdb" containerName="revoke-keys" Nov 02 18:04:36.421650 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.421512 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c8f542-1636-424a-bfcf-8d4aa86e3cdb" containerName="revoke-keys" Nov 02 18:04:36.421650 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.421580 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c8f542-1636-424a-bfcf-8d4aa86e3cdb" containerName="revoke-keys" Nov 02 18:04:36.424980 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.424962 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:36.431093 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.431055 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:04:36.432767 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.432676 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:04:36.437853 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.437782 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff"] Nov 02 18:04:36.480108 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.480059 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-sf4ff\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:36.480284 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.480117 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-sf4ff\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:36.480284 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.480196 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chm8k\" (UniqueName: \"kubernetes.io/projected/8951f289-e47f-4258-987f-a92825dbc0ec-kube-api-access-chm8k\") pod \"maas-api-revoke-sub-ae594979-sf4ff\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:36.581742 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.581704 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-sf4ff\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:36.581742 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.581747 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-sf4ff\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:36.582014 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.581764 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chm8k\" (UniqueName: \"kubernetes.io/projected/8951f289-e47f-4258-987f-a92825dbc0ec-kube-api-access-chm8k\") pod \"maas-api-revoke-sub-ae594979-sf4ff\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:36.582400 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.582375 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-sf4ff\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:36.584146 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.584125 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-sf4ff\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:36.590327 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.590305 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chm8k\" (UniqueName: \"kubernetes.io/projected/8951f289-e47f-4258-987f-a92825dbc0ec-kube-api-access-chm8k\") pod \"maas-api-revoke-sub-ae594979-sf4ff\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:36.739742 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:36.739710 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:37.016242 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:37.016216 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff"] Nov 02 18:04:37.018065 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:04:37.018039 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8951f289_e47f_4258_987f_a92825dbc0ec.slice/crio-f9cc1c7472b47067d0fcfa89b436b30df60f95a7ff83bfde2cd17580046b82b2 WatchSource:0}: Error finding container f9cc1c7472b47067d0fcfa89b436b30df60f95a7ff83bfde2cd17580046b82b2: Status 404 returned error can't find the container with id f9cc1c7472b47067d0fcfa89b436b30df60f95a7ff83bfde2cd17580046b82b2 Nov 02 18:04:37.847786 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:37.847746 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" event={"ID":"8951f289-e47f-4258-987f-a92825dbc0ec","Type":"ContainerStarted","Data":"87c06a8741d463645a2243caadd788f8587c026ca84ec5742ef7a9cf86bc66da"} Nov 02 18:04:37.848212 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:37.847808 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" event={"ID":"8951f289-e47f-4258-987f-a92825dbc0ec","Type":"ContainerStarted","Data":"f9cc1c7472b47067d0fcfa89b436b30df60f95a7ff83bfde2cd17580046b82b2"} Nov 02 18:04:37.869844 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:37.869792 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" podStartSLOduration=1.8697784419999999 podStartE2EDuration="1.869778442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:37.868414166 +0000 UTC m=+1475.311496851" watchObservedRunningTime="2026-11-02 18:04:37.869778442 +0000 UTC m=+1475.312861078" Nov 02 18:04:38.855444 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:38.855404 4310 generic.go:356] "Generic (PLEG): container finished" podID="8951f289-e47f-4258-987f-a92825dbc0ec" containerID="87c06a8741d463645a2243caadd788f8587c026ca84ec5742ef7a9cf86bc66da" exitCode=0 Nov 02 18:04:38.855852 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:38.855503 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" event={"ID":"8951f289-e47f-4258-987f-a92825dbc0ec","Type":"ContainerDied","Data":"87c06a8741d463645a2243caadd788f8587c026ca84ec5742ef7a9cf86bc66da"} Nov 02 18:04:40.000417 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.000386 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:40.118443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.118344 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8951f289-e47f-4258-987f-a92825dbc0ec-kube-api-access-chm8k\" (UniqueName: \"kubernetes.io/projected/8951f289-e47f-4258-987f-a92825dbc0ec-kube-api-access-chm8k\") pod \"8951f289-e47f-4258-987f-a92825dbc0ec\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " Nov 02 18:04:40.118443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.118387 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-openshift-ca\") pod \"8951f289-e47f-4258-987f-a92825dbc0ec\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " Nov 02 18:04:40.118443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.118413 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-xks-ca\") pod \"8951f289-e47f-4258-987f-a92825dbc0ec\" (UID: \"8951f289-e47f-4258-987f-a92825dbc0ec\") " Nov 02 18:04:40.118746 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.118716 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-openshift-ca" pod "8951f289-e47f-4258-987f-a92825dbc0ec" (UID: "8951f289-e47f-4258-987f-a92825dbc0ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:40.118832 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.118813 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.120512 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.120484 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-xks-ca" pod "8951f289-e47f-4258-987f-a92825dbc0ec" (UID: "8951f289-e47f-4258-987f-a92825dbc0ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:40.120621 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.120549 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8951f289-e47f-4258-987f-a92825dbc0ec-kube-api-access-chm8k" pod "8951f289-e47f-4258-987f-a92825dbc0ec" (UID: "8951f289-e47f-4258-987f-a92825dbc0ec"). InnerVolumeSpecName "kube-api-access-chm8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:40.220112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.220040 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chm8k\" (UniqueName: \"kubernetes.io/projected/8951f289-e47f-4258-987f-a92825dbc0ec-kube-api-access-chm8k\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.220112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.220108 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8951f289-e47f-4258-987f-a92825dbc0ec-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.867726 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.867694 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" Nov 02 18:04:40.867923 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.867728 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff" event={"ID":"8951f289-e47f-4258-987f-a92825dbc0ec","Type":"ContainerDied","Data":"f9cc1c7472b47067d0fcfa89b436b30df60f95a7ff83bfde2cd17580046b82b2"} Nov 02 18:04:40.867923 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:40.867778 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9cc1c7472b47067d0fcfa89b436b30df60f95a7ff83bfde2cd17580046b82b2" Nov 02 18:04:48.228623 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:48.228523 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:04:52.041246 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:52.041214 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4"] Nov 02 18:04:52.129793 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:52.129167 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx"] Nov 02 18:04:52.133598 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:52.133562 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe2dba4-jxgg4"] Nov 02 18:04:52.136656 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:52.136626 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18460d9-9fqzx"] Nov 02 18:04:52.673721 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:52.673681 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f6b4ee4-b67f-4c91-9db2-103f7c00c052" path="/var/lib/kubelet/pods/7f6b4ee4-b67f-4c91-9db2-103f7c00c052/volumes" Nov 02 18:04:52.674216 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:52.674195 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa58c510-6474-462b-9455-d676d33e6784" path="/var/lib/kubelet/pods/fa58c510-6474-462b-9455-d676d33e6784/volumes" Nov 02 18:04:54.802236 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.802192 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q"] Nov 02 18:04:54.802754 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.802736 4310 cpu_manager.go:414] "containerMap: removing container" podUID="8951f289-e47f-4258-987f-a92825dbc0ec" containerName="revoke-keys" Nov 02 18:04:54.802800 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.802756 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8951f289-e47f-4258-987f-a92825dbc0ec" containerName="revoke-keys" Nov 02 18:04:54.802836 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.802829 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="8951f289-e47f-4258-987f-a92825dbc0ec" containerName="revoke-keys" Nov 02 18:04:54.805787 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.805773 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:54.811435 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.811413 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:04:54.812511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.812482 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:04:54.819967 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.819943 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q"] Nov 02 18:04:54.866411 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.866378 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-85t6q\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:54.866595 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.866441 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzj7\" (UniqueName: \"kubernetes.io/projected/459f33a3-087a-471b-a572-e74fab1c8341-kube-api-access-rhzj7\") pod \"maas-api-revoke-sub-a4aef7d7-85t6q\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:54.866595 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.866552 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-85t6q\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:54.967254 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.967217 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzj7\" (UniqueName: \"kubernetes.io/projected/459f33a3-087a-471b-a572-e74fab1c8341-kube-api-access-rhzj7\") pod \"maas-api-revoke-sub-a4aef7d7-85t6q\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:54.967432 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.967280 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-85t6q\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:54.967432 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.967349 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-85t6q\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:54.968037 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.968018 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-85t6q\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:54.969806 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.969786 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-85t6q\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:54.975886 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:54.975860 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzj7\" (UniqueName: \"kubernetes.io/projected/459f33a3-087a-471b-a572-e74fab1c8341-kube-api-access-rhzj7\") pod \"maas-api-revoke-sub-a4aef7d7-85t6q\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:55.118766 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:55.118681 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:55.403331 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:55.403300 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q"] Nov 02 18:04:55.404913 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:04:55.404876 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459f33a3_087a_471b_a572_e74fab1c8341.slice/crio-3b9e0d01b58c38b904b6a9b9bbfc6d140ca892eb69c00cae23095eb3050c39d7 WatchSource:0}: Error finding container 3b9e0d01b58c38b904b6a9b9bbfc6d140ca892eb69c00cae23095eb3050c39d7: Status 404 returned error can't find the container with id 3b9e0d01b58c38b904b6a9b9bbfc6d140ca892eb69c00cae23095eb3050c39d7 Nov 02 18:04:55.948320 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:55.948228 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" event={"ID":"459f33a3-087a-471b-a572-e74fab1c8341","Type":"ContainerStarted","Data":"06634541b1a8621320fb4865e515669497939b77ce0165e5790a927d3b6ea34b"} Nov 02 18:04:55.948320 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:55.948280 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" event={"ID":"459f33a3-087a-471b-a572-e74fab1c8341","Type":"ContainerStarted","Data":"3b9e0d01b58c38b904b6a9b9bbfc6d140ca892eb69c00cae23095eb3050c39d7"} Nov 02 18:04:55.972689 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:55.972630 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" podStartSLOduration=1.972616312 podStartE2EDuration="1.972616312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:55.969922353 +0000 UTC m=+1493.413004990" watchObservedRunningTime="2026-11-02 18:04:55.972616312 +0000 UTC m=+1493.415698975" Nov 02 18:04:56.954799 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:56.954767 4310 generic.go:356] "Generic (PLEG): container finished" podID="459f33a3-087a-471b-a572-e74fab1c8341" containerID="06634541b1a8621320fb4865e515669497939b77ce0165e5790a927d3b6ea34b" exitCode=0 Nov 02 18:04:56.955229 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:56.954862 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" event={"ID":"459f33a3-087a-471b-a572-e74fab1c8341","Type":"ContainerDied","Data":"06634541b1a8621320fb4865e515669497939b77ce0165e5790a927d3b6ea34b"} Nov 02 18:04:58.097275 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.097240 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:58.198679 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.198639 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-xks-ca\") pod \"459f33a3-087a-471b-a572-e74fab1c8341\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " Nov 02 18:04:58.198859 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.198734 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-openshift-ca\") pod \"459f33a3-087a-471b-a572-e74fab1c8341\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " Nov 02 18:04:58.198859 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.198782 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459f33a3-087a-471b-a572-e74fab1c8341-kube-api-access-rhzj7\" (UniqueName: \"kubernetes.io/projected/459f33a3-087a-471b-a572-e74fab1c8341-kube-api-access-rhzj7\") pod \"459f33a3-087a-471b-a572-e74fab1c8341\" (UID: \"459f33a3-087a-471b-a572-e74fab1c8341\") " Nov 02 18:04:58.199146 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.199112 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-openshift-ca" pod "459f33a3-087a-471b-a572-e74fab1c8341" (UID: "459f33a3-087a-471b-a572-e74fab1c8341"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:58.200761 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.200731 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-xks-ca" pod "459f33a3-087a-471b-a572-e74fab1c8341" (UID: "459f33a3-087a-471b-a572-e74fab1c8341"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:58.200855 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.200828 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459f33a3-087a-471b-a572-e74fab1c8341-kube-api-access-rhzj7" pod "459f33a3-087a-471b-a572-e74fab1c8341" (UID: "459f33a3-087a-471b-a572-e74fab1c8341"). InnerVolumeSpecName "kube-api-access-rhzj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:58.300365 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.300329 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:04:58.300365 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.300358 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhzj7\" (UniqueName: \"kubernetes.io/projected/459f33a3-087a-471b-a572-e74fab1c8341-kube-api-access-rhzj7\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:04:58.300365 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.300368 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459f33a3-087a-471b-a572-e74fab1c8341-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:04:58.967300 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.967269 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" Nov 02 18:04:58.967492 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.967273 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q" event={"ID":"459f33a3-087a-471b-a572-e74fab1c8341","Type":"ContainerDied","Data":"3b9e0d01b58c38b904b6a9b9bbfc6d140ca892eb69c00cae23095eb3050c39d7"} Nov 02 18:04:58.967492 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:04:58.967403 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b9e0d01b58c38b904b6a9b9bbfc6d140ca892eb69c00cae23095eb3050c39d7" Nov 02 18:05:03.523234 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:05:03.523191 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:05:11.981623 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:11.981584 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:13.720474 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.720442 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl"] Nov 02 18:05:13.720890 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.720863 4310 cpu_manager.go:414] "containerMap: removing container" podUID="459f33a3-087a-471b-a572-e74fab1c8341" containerName="revoke-keys" Nov 02 18:05:13.720890 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.720875 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459f33a3-087a-471b-a572-e74fab1c8341" containerName="revoke-keys" Nov 02 18:05:13.720983 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.720932 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="459f33a3-087a-471b-a572-e74fab1c8341" containerName="revoke-keys" Nov 02 18:05:13.725462 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.725436 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:13.733035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.733007 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:05:13.734053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.734024 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:05:13.738184 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.738159 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl"] Nov 02 18:05:13.759167 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.759136 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bbtbl\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:13.759347 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.759179 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-666jx\" (UniqueName: \"kubernetes.io/projected/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-kube-api-access-666jx\") pod \"maas-api-revoke-sub-79805ca9-bbtbl\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:13.759413 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.759342 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bbtbl\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:13.860846 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.860788 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bbtbl\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:13.861070 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.860884 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bbtbl\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:13.861070 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.860921 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-666jx\" (UniqueName: \"kubernetes.io/projected/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-kube-api-access-666jx\") pod \"maas-api-revoke-sub-79805ca9-bbtbl\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:13.863021 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.862099 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bbtbl\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:13.864517 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.864487 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bbtbl\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:13.891879 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:13.891844 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-666jx\" (UniqueName: \"kubernetes.io/projected/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-kube-api-access-666jx\") pod \"maas-api-revoke-sub-79805ca9-bbtbl\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:14.040889 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:14.040782 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:14.319040 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:14.319012 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl"] Nov 02 18:05:14.320123 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:05:14.320072 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb735cfa2_f6a4_4c1d_8aa1_681ecdd3f5f7.slice/crio-4c453eee8a67ac1c27b4f9cf117197e9356f547f7e0d5440ae63f02d2919aeca WatchSource:0}: Error finding container 4c453eee8a67ac1c27b4f9cf117197e9356f547f7e0d5440ae63f02d2919aeca: Status 404 returned error can't find the container with id 4c453eee8a67ac1c27b4f9cf117197e9356f547f7e0d5440ae63f02d2919aeca Nov 02 18:05:15.051951 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:15.051911 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" event={"ID":"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7","Type":"ContainerStarted","Data":"41fa412b7c2c2979efcdc12e4efd77eb2f0a5d0ff0a703096c3500f939a92da9"} Nov 02 18:05:15.052430 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:15.051962 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" event={"ID":"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7","Type":"ContainerStarted","Data":"4c453eee8a67ac1c27b4f9cf117197e9356f547f7e0d5440ae63f02d2919aeca"} Nov 02 18:05:15.092950 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:15.092898 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" podStartSLOduration=2.092881547 podStartE2EDuration="2.092881547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:15.092882117 +0000 UTC m=+1512.535964753" watchObservedRunningTime="2026-11-02 18:05:15.092881547 +0000 UTC m=+1512.535964183" Nov 02 18:05:16.059038 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:16.059001 4310 generic.go:356] "Generic (PLEG): container finished" podID="b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7" containerID="41fa412b7c2c2979efcdc12e4efd77eb2f0a5d0ff0a703096c3500f939a92da9" exitCode=0 Nov 02 18:05:16.059490 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:16.059103 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" event={"ID":"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7","Type":"ContainerDied","Data":"41fa412b7c2c2979efcdc12e4efd77eb2f0a5d0ff0a703096c3500f939a92da9"} Nov 02 18:05:17.200743 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:17.200712 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:17.293767 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:17.293736 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-openshift-ca\") pod \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " Nov 02 18:05:17.293950 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:17.293773 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-kube-api-access-666jx\" (UniqueName: \"kubernetes.io/projected/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-kube-api-access-666jx\") pod \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " Nov 02 18:05:17.293950 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:17.293814 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-xks-ca\") pod \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\" (UID: \"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7\") " Nov 02 18:05:17.294211 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:17.294183 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-openshift-ca" pod "b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7" (UID: "b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:17.295906 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:17.295883 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-xks-ca" pod "b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7" (UID: "b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:17.295991 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:17.295954 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-kube-api-access-666jx" pod "b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7" (UID: "b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7"). InnerVolumeSpecName "kube-api-access-666jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:17.395508 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:17.395430 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-666jx\" (UniqueName: \"kubernetes.io/projected/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-kube-api-access-666jx\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:17.395508 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:17.395465 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:17.395508 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:17.395476 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:05:18.071334 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:18.071304 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" Nov 02 18:05:18.071511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:18.071344 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl" event={"ID":"b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7","Type":"ContainerDied","Data":"4c453eee8a67ac1c27b4f9cf117197e9356f547f7e0d5440ae63f02d2919aeca"} Nov 02 18:05:18.071511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:18.071443 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c453eee8a67ac1c27b4f9cf117197e9356f547f7e0d5440ae63f02d2919aeca" Nov 02 18:05:29.160102 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:29.158285 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-l98w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:05:29.160102 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:29.158401 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-l98w5" podUID="d1be36c1-1e0e-478d-bdeb-7a9704cee2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:05:31.828024 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:31.827985 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:33.522804 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:05:33.522765 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:05:40.737898 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:40.737862 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:40.847751 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:40.847712 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:05:48.274179 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:05:48.274146 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:03.522888 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:06:03.522848 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:06:05.040249 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:05.040201 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:05.168497 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:05.168464 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:05.308396 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:05.308320 4310 scope.go:122] "RemoveContainer" containerID="7c06fcc4758b3d3cd2463f21af3d6ec64bf724ac32121aaf68236c74fca859d9" Nov 02 18:06:05.317050 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:05.317027 4310 scope.go:122] "RemoveContainer" containerID="f9d47020381e4741480563b9ecf03208cd9b8c6c2c82b449dc566963afb26b46" Nov 02 18:06:05.325121 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:05.325099 4310 scope.go:122] "RemoveContainer" containerID="2b14b468375c7de31e6bf1c9b9179e16327ca24c141e5009e139afe1455579a9" Nov 02 18:06:18.031311 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:18.031211 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr"] Nov 02 18:06:18.034250 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:18.034219 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd"] Nov 02 18:06:18.039927 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:18.039898 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436ac4ab-dztfr"] Nov 02 18:06:18.041829 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:18.041805 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b2efea-prmxd"] Nov 02 18:06:18.672776 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:18.672740 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357f7f96-2e25-4fb2-b836-7ff645d55283" path="/var/lib/kubelet/pods/357f7f96-2e25-4fb2-b836-7ff645d55283/volumes" Nov 02 18:06:18.673196 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:18.673182 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c9d32a-137e-444b-99a1-bcec773e85d5" path="/var/lib/kubelet/pods/48c9d32a-137e-444b-99a1-bcec773e85d5/volumes" Nov 02 18:06:21.257907 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:21.257873 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:21.385394 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:21.385354 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:21.848026 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:21.847991 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:21.969164 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:21.969110 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:06:33.522691 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:06:33.522652 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:06:45.025291 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:45.025244 4310 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkwgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:06:45.025806 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:45.025305 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkwgh" podUID="1a783ce1-3916-4723-a400-9155ea747d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:06:48.324468 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:06:48.324429 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:03.522682 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:07:03.522642 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:07:05.351047 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:05.351014 4310 scope.go:122] "RemoveContainer" containerID="fa786e7120ea527679229304598fe9c41557513560937f404c47cc42b545b16a" Nov 02 18:07:05.359488 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:05.359466 4310 scope.go:122] "RemoveContainer" containerID="8380e2ae66283467884a17c6319fdfa2b24bee5b3ea8e3b5f37c85feeb98b947" Nov 02 18:07:07.021207 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:07.021178 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7"] Nov 02 18:07:07.024072 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:07.024042 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xbdk7"] Nov 02 18:07:08.673121 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:08.673065 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9d464b-a820-45cc-a0ba-eb13fc6b9419" path="/var/lib/kubelet/pods/6a9d464b-a820-45cc-a0ba-eb13fc6b9419/volumes" Nov 02 18:07:12.215411 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.215351 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:12.620641 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.620560 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h"] Nov 02 18:07:12.621218 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.621202 4310 cpu_manager.go:414] "containerMap: removing container" podUID="b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7" containerName="revoke-keys" Nov 02 18:07:12.621273 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.621221 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7" containerName="revoke-keys" Nov 02 18:07:12.621349 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.621339 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7" containerName="revoke-keys" Nov 02 18:07:12.626393 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.626369 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.636988 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.636958 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-836cee-openshift-default-dockercfg-cv9jw\"" Nov 02 18:07:12.637518 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.637492 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h"] Nov 02 18:07:12.665529 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.665476 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:12.667671 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.667643 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-token\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.667837 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.667758 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-data\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.667837 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.667794 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-podinfo\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.667979 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.667928 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-socket\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.668094 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.668020 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-envoy\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.668172 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.668113 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-certs\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.668172 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.668163 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad861f-f9e3-4422-b663-917fafe4df16-istiod-ca-cert\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.668290 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.668229 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-credential-socket\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.668290 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.668272 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smdt7\" (UniqueName: \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-kube-api-access-smdt7\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.769652 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.769619 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-credential-socket\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.769856 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.769671 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smdt7\" (UniqueName: \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-kube-api-access-smdt7\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.769856 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.769712 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-token\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.769856 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.769744 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-data\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.770115 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.769869 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-podinfo\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.770115 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.769925 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-socket\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.770115 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.769953 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-envoy\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.770115 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.770011 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-certs\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.770115 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.770024 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-credential-socket\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.770115 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.770064 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad861f-f9e3-4422-b663-917fafe4df16-istiod-ca-cert\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.770115 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.770073 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-data\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.770538 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.770301 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-socket\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.770538 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.770335 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-certs\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.770748 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.770728 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad861f-f9e3-4422-b663-917fafe4df16-istiod-ca-cert\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.772247 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.772226 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-envoy\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.772457 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.772439 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-podinfo\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.778258 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.778232 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smdt7\" (UniqueName: \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-kube-api-access-smdt7\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.779137 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.779071 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:12.779268 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.779250 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-token\") pod \"e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:12.940493 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:12.940411 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:13.218840 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:13.218812 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h"] Nov 02 18:07:13.220831 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:07:13.220803 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ad861f_f9e3_4422_b663_917fafe4df16.slice/crio-5eec78f2217b6f05ac63bac67724a70600b8f20fae282d61ebe3614965624c1b WatchSource:0}: Error finding container 5eec78f2217b6f05ac63bac67724a70600b8f20fae282d61ebe3614965624c1b: Status 404 returned error can't find the container with id 5eec78f2217b6f05ac63bac67724a70600b8f20fae282d61ebe3614965624c1b Nov 02 18:07:13.222923 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:13.222891 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:07:13.223023 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:13.222969 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:07:13.223023 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:13.223012 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:07:13.771549 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:13.771510 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" event={"ID":"f5ad861f-f9e3-4422-b663-917fafe4df16","Type":"ContainerStarted","Data":"87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3"} Nov 02 18:07:13.771716 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:13.771575 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" event={"ID":"f5ad861f-f9e3-4422-b663-917fafe4df16","Type":"ContainerStarted","Data":"5eec78f2217b6f05ac63bac67724a70600b8f20fae282d61ebe3614965624c1b"} Nov 02 18:07:13.798061 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:13.798013 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" podStartSLOduration=1.7979974749999998 podStartE2EDuration="1.797997475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:13.795392731 +0000 UTC m=+1631.238475382" watchObservedRunningTime="2026-11-02 18:07:13.797997475 +0000 UTC m=+1631.241080110" Nov 02 18:07:13.940565 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:13.940534 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:14.946678 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:14.946648 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:15.781958 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:15.781910 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:15.783147 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:15.783123 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:16.055564 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:16.055470 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:16.357060 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:16.356968 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:17.253469 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:17.253431 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:17.396896 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:17.396860 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:18.089358 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:18.089322 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:19.000784 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.000751 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:19.726519 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.726475 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:19.791908 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.791874 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4"] Nov 02 18:07:19.795688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.795670 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:07:19.802904 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.802878 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-836cee\"" Nov 02 18:07:19.804395 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.804371 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-1637c359-dockercfg-wf29k\"" Nov 02 18:07:19.806663 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.806637 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4"] Nov 02 18:07:19.843288 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.843260 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:19.845406 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.845375 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1641d288-9f61-49e4-8119-322785ca136c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4\" (UID: \"1641d288-9f61-49e4-8119-322785ca136c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:07:19.846033 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.845441 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8w2p\" (UniqueName: \"kubernetes.io/projected/1641d288-9f61-49e4-8119-322785ca136c-kube-api-access-r8w2p\") pod \"payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4\" (UID: \"1641d288-9f61-49e4-8119-322785ca136c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:07:19.948023 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.947978 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1641d288-9f61-49e4-8119-322785ca136c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4\" (UID: \"1641d288-9f61-49e4-8119-322785ca136c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:07:19.948274 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.948045 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8w2p\" (UniqueName: \"kubernetes.io/projected/1641d288-9f61-49e4-8119-322785ca136c-kube-api-access-r8w2p\") pod \"payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4\" (UID: \"1641d288-9f61-49e4-8119-322785ca136c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:07:19.948998 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.948966 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1641d288-9f61-49e4-8119-322785ca136c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4\" (UID: \"1641d288-9f61-49e4-8119-322785ca136c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:07:19.958114 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:19.958048 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8w2p\" (UniqueName: \"kubernetes.io/projected/1641d288-9f61-49e4-8119-322785ca136c-kube-api-access-r8w2p\") pod \"payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4\" (UID: \"1641d288-9f61-49e4-8119-322785ca136c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:07:20.112647 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:20.112558 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:07:20.410318 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:20.410282 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4"] Nov 02 18:07:20.411553 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:07:20.411529 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1641d288_9f61_49e4_8119_322785ca136c.slice/crio-adfc64eb89b42df067bd71d14f3ab1654577bd120f06a63f5311925fc3c1a904 WatchSource:0}: Error finding container adfc64eb89b42df067bd71d14f3ab1654577bd120f06a63f5311925fc3c1a904: Status 404 returned error can't find the container with id adfc64eb89b42df067bd71d14f3ab1654577bd120f06a63f5311925fc3c1a904 Nov 02 18:07:20.584335 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:20.584306 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:20.811504 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:20.811463 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" event={"ID":"1641d288-9f61-49e4-8119-322785ca136c","Type":"ContainerStarted","Data":"9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a"} Nov 02 18:07:20.811730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:20.811523 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" event={"ID":"1641d288-9f61-49e4-8119-322785ca136c","Type":"ContainerStarted","Data":"adfc64eb89b42df067bd71d14f3ab1654577bd120f06a63f5311925fc3c1a904"} Nov 02 18:07:20.811730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:20.811677 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:07:20.830884 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:20.830836 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" podStartSLOduration=1.830822456 podStartE2EDuration="1.830822456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:20.829529866 +0000 UTC m=+1638.272612504" watchObservedRunningTime="2026-11-02 18:07:20.830822456 +0000 UTC m=+1638.273905092" Nov 02 18:07:21.400395 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:21.400356 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:22.289898 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:22.289865 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:22.995368 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:22.995331 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:23.735425 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:23.735395 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:23.836813 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:23.836780 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:23.944323 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:23.940144 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:24.589829 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:24.589792 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:25.290677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:25.290641 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:26.055249 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:26.055213 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:26.790414 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:26.790354 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:27.464643 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:27.464607 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:28.160011 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:28.159971 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:28.860396 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:28.860364 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:29.476930 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:29.476864 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:29.865693 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:29.865613 4310 generic.go:356] "Generic (PLEG): container finished" podID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerID="87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3" exitCode=0 Nov 02 18:07:29.866067 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:29.865688 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" event={"ID":"f5ad861f-f9e3-4422-b663-917fafe4df16","Type":"ContainerDied","Data":"87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3"} Nov 02 18:07:29.866173 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:29.866150 4310 scope.go:122] "RemoveContainer" containerID="87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3" Nov 02 18:07:29.867401 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:29.867365 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:07:29.867499 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:29.867431 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:07:29.867499 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:29.867465 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:07:30.010833 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:30.010804 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:30.173363 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:30.173253 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:30.782359 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:30.782321 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:30.881012 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:30.880975 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" event={"ID":"f5ad861f-f9e3-4422-b663-917fafe4df16","Type":"ContainerStarted","Data":"878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb"} Nov 02 18:07:30.896479 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:30.896443 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:30.946707 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:30.946662 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:30.949296 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:30.949255 4310 patch_prober.go:29] interesting pod/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 18:07:30.949602 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:30.949554 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 18:07:31.590554 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:31.590517 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:31.819513 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:31.819478 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:07:31.941475 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:31.941379 4310 patch_prober.go:29] interesting pod/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 18:07:31.941475 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:31.941459 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 18:07:32.423322 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:32.423284 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:32.940907 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:32.940877 4310 patch_prober.go:29] interesting pod/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 18:07:32.941121 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:32.940931 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 18:07:33.159705 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:33.159664 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:33.524035 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:07:33.523997 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:07:33.874442 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:33.874357 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:33.942887 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:33.942452 4310 patch_prober.go:29] interesting pod/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 18:07:33.942887 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:33.942510 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 18:07:34.684247 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:34.684218 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:34.940894 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:34.940809 4310 patch_prober.go:29] interesting pod/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 18:07:34.940894 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:34.940865 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 18:07:35.375109 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:35.375038 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:36.014119 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:36.010272 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:36.014119 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:36.010832 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:36.014119 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:36.014134 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:07:36.068296 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:36.068260 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:36.755058 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:36.755025 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:37.373033 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:37.373000 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:38.049063 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:38.049030 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:38.678133 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:38.678091 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:39.360368 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:39.360322 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:40.054510 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:40.054476 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:40.762440 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:40.762406 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:41.463298 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:41.463266 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:42.113751 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:42.113708 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:42.752098 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:42.752055 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:43.082804 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:43.082725 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:43.563363 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:43.563327 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:43.765735 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:43.765681 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:43.864528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:43.864428 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:44.443902 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:44.443860 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:45.001593 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:45.001560 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:45.282061 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:45.281975 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:46.167920 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:46.167888 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:47.000176 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:47.000143 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:47.782275 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:47.782245 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:48.354314 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:48.354274 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:48.600209 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:48.600175 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:49.299841 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:49.299808 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:49.969178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:49.969144 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:50.648591 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:50.648557 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:51.303035 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:51.303002 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:51.960648 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:51.960617 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:52.766242 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:52.766201 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:53.462917 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:53.462882 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:54.181281 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:54.181247 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:54.883931 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:54.883900 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:55.563487 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:55.563435 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:56.175520 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:56.175483 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:56.998385 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:56.998356 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:57.696660 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:57.696625 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:58.392015 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:58.391976 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:59.072466 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:59.072436 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:07:59.761835 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:07:59.761797 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:00.470549 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:00.470510 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:01.023488 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:01.023436 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg"] Nov 02 18:08:01.025748 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:01.025721 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xtdsg"] Nov 02 18:08:01.267859 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:01.267827 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:01.973457 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:01.973420 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:02.662132 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:02.662099 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:02.673661 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:02.673630 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d51cd7f-a7a5-4803-b6c6-981fb0831b12" path="/var/lib/kubelet/pods/5d51cd7f-a7a5-4803-b6c6-981fb0831b12/volumes" Nov 02 18:08:03.462184 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:03.462150 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:03.539157 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:08:03.539111 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:08:04.261139 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:04.261108 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:04.879930 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:04.879881 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:05.384535 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:05.384505 4310 scope.go:122] "RemoveContainer" containerID="33b79af25383f5255cd636e9957ddd81513fb223457e15ae2967d8df12a57699" Nov 02 18:08:05.394871 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:05.394851 4310 scope.go:122] "RemoveContainer" containerID="2e722dbc88531ef3020c2933ae2d5aa05b80b692ae1dccee3f058dc9681c4a72" Nov 02 18:08:05.404288 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:05.404266 4310 scope.go:122] "RemoveContainer" containerID="3a3038ebdd5e73c11a13490fbf187345ba82a1763a4720c5b07c76eac2357984" Nov 02 18:08:05.553686 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:05.553655 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:06.291792 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:06.291759 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:06.961888 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:06.961856 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:07.653339 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:07.653306 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:08.366113 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:08.366048 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:09.061497 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:09.061462 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:09.685869 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:09.685836 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:10.379823 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:10.379789 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:11.080805 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:11.080770 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:11.692097 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:11.692046 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:12.486809 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:12.486777 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:13.155901 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:13.155866 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:13.768048 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:13.768002 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:14.449768 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:14.449736 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:14.845752 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:14.845719 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b"] Nov 02 18:08:14.850815 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:14.850794 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:14.855825 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:14.855798 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:08:14.855975 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:14.855848 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:08:14.859061 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:14.858825 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b"] Nov 02 18:08:14.935926 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:14.935880 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc483f5f-fjl4b\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:14.936178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:14.935942 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jlr5\" (UniqueName: \"kubernetes.io/projected/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-kube-api-access-6jlr5\") pod \"maas-api-revoke-sub-cc483f5f-fjl4b\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:14.936178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:14.936166 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc483f5f-fjl4b\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:15.037449 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:15.037408 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc483f5f-fjl4b\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:15.037658 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:15.037461 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc483f5f-fjl4b\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:15.037658 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:15.037488 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlr5\" (UniqueName: \"kubernetes.io/projected/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-kube-api-access-6jlr5\") pod \"maas-api-revoke-sub-cc483f5f-fjl4b\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:15.038072 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:15.038049 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc483f5f-fjl4b\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:15.039838 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:15.039813 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc483f5f-fjl4b\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:15.046141 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:15.046119 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jlr5\" (UniqueName: \"kubernetes.io/projected/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-kube-api-access-6jlr5\") pod \"maas-api-revoke-sub-cc483f5f-fjl4b\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:15.167604 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:15.167520 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:15.269751 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:15.269711 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:15.445334 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:15.445302 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b"] Nov 02 18:08:15.447646 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:08:15.447620 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d948f6_3fb1_4a14_addb_a36c7e95a2f7.slice/crio-6d8b9eaffc5550f5905bad3a3f6b249f8d583edc11cbd0b34fdfbfe7835d0a91 WatchSource:0}: Error finding container 6d8b9eaffc5550f5905bad3a3f6b249f8d583edc11cbd0b34fdfbfe7835d0a91: Status 404 returned error can't find the container with id 6d8b9eaffc5550f5905bad3a3f6b249f8d583edc11cbd0b34fdfbfe7835d0a91 Nov 02 18:08:15.961462 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:15.961425 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:16.123180 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:16.123063 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" event={"ID":"22d948f6-3fb1-4a14-addb-a36c7e95a2f7","Type":"ContainerStarted","Data":"fc7af6e354ea1c0ed78bc1ce35623e2d9c03a2af1ef95386ae125b804ba4a313"} Nov 02 18:08:16.123180 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:16.123150 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" event={"ID":"22d948f6-3fb1-4a14-addb-a36c7e95a2f7","Type":"ContainerStarted","Data":"6d8b9eaffc5550f5905bad3a3f6b249f8d583edc11cbd0b34fdfbfe7835d0a91"} Nov 02 18:08:16.139415 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:16.139349 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" podStartSLOduration=2.139330163 podStartE2EDuration="2.139330163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:16.138669634 +0000 UTC m=+1693.581752270" watchObservedRunningTime="2026-11-02 18:08:16.139330163 +0000 UTC m=+1693.582412799" Nov 02 18:08:16.644160 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:16.644127 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:17.130052 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:17.130010 4310 generic.go:356] "Generic (PLEG): container finished" podID="22d948f6-3fb1-4a14-addb-a36c7e95a2f7" containerID="fc7af6e354ea1c0ed78bc1ce35623e2d9c03a2af1ef95386ae125b804ba4a313" exitCode=0 Nov 02 18:08:17.130514 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:17.130123 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" event={"ID":"22d948f6-3fb1-4a14-addb-a36c7e95a2f7","Type":"ContainerDied","Data":"fc7af6e354ea1c0ed78bc1ce35623e2d9c03a2af1ef95386ae125b804ba4a313"} Nov 02 18:08:17.392553 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:17.392450 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:18.058859 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.058826 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:18.275418 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.275389 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:18.319924 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.319842 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:18.369710 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.369681 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-kube-api-access-6jlr5\" (UniqueName: \"kubernetes.io/projected/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-kube-api-access-6jlr5\") pod \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " Nov 02 18:08:18.369913 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.369718 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-openshift-ca\") pod \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " Nov 02 18:08:18.369913 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.369865 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-xks-ca\") pod \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\" (UID: \"22d948f6-3fb1-4a14-addb-a36c7e95a2f7\") " Nov 02 18:08:18.370095 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.370060 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-openshift-ca" pod "22d948f6-3fb1-4a14-addb-a36c7e95a2f7" (UID: "22d948f6-3fb1-4a14-addb-a36c7e95a2f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:18.370328 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.370309 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:18.371908 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.371886 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-kube-api-access-6jlr5" pod "22d948f6-3fb1-4a14-addb-a36c7e95a2f7" (UID: "22d948f6-3fb1-4a14-addb-a36c7e95a2f7"). InnerVolumeSpecName "kube-api-access-6jlr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:18.372003 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.371913 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-xks-ca" pod "22d948f6-3fb1-4a14-addb-a36c7e95a2f7" (UID: "22d948f6-3fb1-4a14-addb-a36c7e95a2f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:08:18.417346 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.417319 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:18.470807 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.470775 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:18.470807 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.470805 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jlr5\" (UniqueName: \"kubernetes.io/projected/22d948f6-3fb1-4a14-addb-a36c7e95a2f7-kube-api-access-6jlr5\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:18.894868 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:18.894836 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:19.147142 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:19.147111 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" Nov 02 18:08:19.147142 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:19.147130 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b" event={"ID":"22d948f6-3fb1-4a14-addb-a36c7e95a2f7","Type":"ContainerDied","Data":"6d8b9eaffc5550f5905bad3a3f6b249f8d583edc11cbd0b34fdfbfe7835d0a91"} Nov 02 18:08:19.147371 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:19.147186 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d8b9eaffc5550f5905bad3a3f6b249f8d583edc11cbd0b34fdfbfe7835d0a91" Nov 02 18:08:19.682332 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:19.682297 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:20.390804 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:20.390769 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:21.080425 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:21.080390 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:21.784269 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:21.784231 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:22.039836 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:22.039752 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj"] Nov 02 18:08:22.044994 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:22.044966 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lggcj"] Nov 02 18:08:22.476995 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:22.476956 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:22.673358 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:22.673319 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5237c66-6b6e-4279-88c4-e826ed10d598" path="/var/lib/kubelet/pods/a5237c66-6b6e-4279-88c4-e826ed10d598/volumes" Nov 02 18:08:23.167328 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:23.167297 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:24.011849 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:24.011808 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:24.749454 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:24.748173 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:25.363898 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.363854 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:25.701598 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.701520 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:25.848665 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.848631 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm"] Nov 02 18:08:25.849071 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.849059 4310 cpu_manager.go:414] "containerMap: removing container" podUID="22d948f6-3fb1-4a14-addb-a36c7e95a2f7" containerName="revoke-keys" Nov 02 18:08:25.849141 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.849073 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d948f6-3fb1-4a14-addb-a36c7e95a2f7" containerName="revoke-keys" Nov 02 18:08:25.849178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.849163 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d948f6-3fb1-4a14-addb-a36c7e95a2f7" containerName="revoke-keys" Nov 02 18:08:25.852281 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.852264 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:25.857027 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.856999 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:08:25.857822 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.857799 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:08:25.862744 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.862543 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm"] Nov 02 18:08:25.956831 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.956744 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:25.956831 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.956783 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhlk\" (UniqueName: \"kubernetes.io/projected/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-kube-api-access-ghhlk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:25.956831 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:25.956808 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:26.058242 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:26.058206 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:26.058455 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:26.058247 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhlk\" (UniqueName: \"kubernetes.io/projected/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-kube-api-access-ghhlk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:26.058455 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:26.058270 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:26.059150 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:26.059124 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:26.060837 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:26.060813 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:26.067978 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:26.067951 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhlk\" (UniqueName: \"kubernetes.io/projected/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-kube-api-access-ghhlk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:26.172021 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:26.171988 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:26.339476 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:26.339437 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:26.447482 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:26.447451 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm"] Nov 02 18:08:26.448616 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:08:26.448591 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf91ddd_1aef_40f2_aee8_0dc7013fc2fd.slice/crio-2bdc46988ce29aa00da75b122b5c0e721239b9cb2328d5ed46ca2b4d6796416a WatchSource:0}: Error finding container 2bdc46988ce29aa00da75b122b5c0e721239b9cb2328d5ed46ca2b4d6796416a: Status 404 returned error can't find the container with id 2bdc46988ce29aa00da75b122b5c0e721239b9cb2328d5ed46ca2b4d6796416a Nov 02 18:08:27.186764 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:27.186649 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:27.190203 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:27.190175 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" event={"ID":"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd","Type":"ContainerStarted","Data":"a9619da3aaf158c000e8f392ab3f9a334bc1310a6f6d9034d9567c58d3b92228"} Nov 02 18:08:27.190384 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:27.190224 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" event={"ID":"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd","Type":"ContainerStarted","Data":"2bdc46988ce29aa00da75b122b5c0e721239b9cb2328d5ed46ca2b4d6796416a"} Nov 02 18:08:27.210484 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:27.210434 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" podStartSLOduration=2.210422158 podStartE2EDuration="2.210422158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:27.208985336 +0000 UTC m=+1704.652067973" watchObservedRunningTime="2026-11-02 18:08:27.210422158 +0000 UTC m=+1704.653504794" Nov 02 18:08:28.022512 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:28.022182 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:28.196818 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:28.196788 4310 generic.go:356] "Generic (PLEG): container finished" podID="8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd" containerID="a9619da3aaf158c000e8f392ab3f9a334bc1310a6f6d9034d9567c58d3b92228" exitCode=0 Nov 02 18:08:28.196999 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:28.196888 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" event={"ID":"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd","Type":"ContainerDied","Data":"a9619da3aaf158c000e8f392ab3f9a334bc1310a6f6d9034d9567c58d3b92228"} Nov 02 18:08:28.656951 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:28.656918 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:29.344998 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:29.344968 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:29.443229 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:29.443180 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:29.494657 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:29.494618 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-kube-api-access-ghhlk\" (UniqueName: \"kubernetes.io/projected/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-kube-api-access-ghhlk\") pod \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " Nov 02 18:08:29.494878 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:29.494680 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-openshift-ca\") pod \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " Nov 02 18:08:29.494878 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:29.494800 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-xks-ca\") pod \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\" (UID: \"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd\") " Nov 02 18:08:29.495173 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:29.495140 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-openshift-ca" pod "8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd" (UID: "8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:29.496909 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:29.496884 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-kube-api-access-ghhlk" pod "8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd" (UID: "8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd"). InnerVolumeSpecName "kube-api-access-ghhlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:29.496975 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:29.496938 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-xks-ca" pod "8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd" (UID: "8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:08:29.596425 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:29.596388 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghhlk\" (UniqueName: \"kubernetes.io/projected/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-kube-api-access-ghhlk\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:29.596425 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:29.596417 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:29.596425 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:29.596430 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:30.078992 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:30.078949 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:30.209377 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:30.209332 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" event={"ID":"8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd","Type":"ContainerDied","Data":"2bdc46988ce29aa00da75b122b5c0e721239b9cb2328d5ed46ca2b4d6796416a"} Nov 02 18:08:30.209557 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:30.209355 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm" Nov 02 18:08:30.209557 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:30.209439 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bdc46988ce29aa00da75b122b5c0e721239b9cb2328d5ed46ca2b4d6796416a" Nov 02 18:08:30.738766 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:30.738732 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:30.869452 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:30.869411 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:31.572993 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:31.572955 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:32.258951 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:32.258913 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:32.967046 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:32.967010 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:33.024551 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:33.024517 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn"] Nov 02 18:08:33.026928 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:33.026891 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zj4vn"] Nov 02 18:08:33.523202 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:08:33.523166 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:08:33.658793 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:33.658758 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:34.363200 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:34.363160 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:34.673841 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:34.673768 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a112456-8204-4567-871e-5be0fe6a7e7e" path="/var/lib/kubelet/pods/9a112456-8204-4567-871e-5be0fe6a7e7e/volumes" Nov 02 18:08:35.065400 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:35.065363 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:35.801277 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:35.801239 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:35.990421 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:35.990383 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4"] Nov 02 18:08:36.057173 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:36.057057 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:36.153736 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:36.153705 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:36.477574 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:36.477542 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:37.261922 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:37.261889 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm"] Nov 02 18:08:37.266261 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:37.266228 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-836cee-j6qkm"] Nov 02 18:08:37.908869 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:37.908835 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:38.672180 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:38.672141 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd" path="/var/lib/kubelet/pods/8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd/volumes" Nov 02 18:08:40.991376 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:40.991327 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" podUID="1641d288-9f61-49e4-8119-322785ca136c" containerName="payload-pre-processing" containerID="cri-o://9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a" gracePeriod=25 Nov 02 18:08:41.817890 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:41.817847 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" podUID="1641d288-9f61-49e4-8119-322785ca136c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:08:43.932056 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:43.932007 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:44.039880 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:44.039833 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:44.256209 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:44.256173 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h"] Nov 02 18:08:44.256463 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:44.256441 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" containerID="cri-o://878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb" gracePeriod=30 Nov 02 18:08:44.304209 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:44.304156 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:45.522004 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.521962 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c"] Nov 02 18:08:45.522459 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.522449 4310 cpu_manager.go:414] "containerMap: removing container" podUID="8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd" containerName="revoke-keys" Nov 02 18:08:45.522514 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.522461 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd" containerName="revoke-keys" Nov 02 18:08:45.522554 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.522539 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf91ddd-1aef-40f2-aee8-0dc7013fc2fd" containerName="revoke-keys" Nov 02 18:08:45.526826 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.526807 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:45.532639 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.532608 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:08:45.532850 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.532669 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:08:45.535977 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.535948 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c"] Nov 02 18:08:45.557940 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.557904 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3850aa45-c7d3-4367-883b-177428028837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k7b9c\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:45.558170 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.557952 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdb25\" (UniqueName: \"kubernetes.io/projected/3850aa45-c7d3-4367-883b-177428028837-kube-api-access-vdb25\") pod \"maas-api-revoke-sub-3d4156f1-k7b9c\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:45.558170 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.558140 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3850aa45-c7d3-4367-883b-177428028837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k7b9c\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:45.659516 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.659459 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3850aa45-c7d3-4367-883b-177428028837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k7b9c\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:45.659707 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.659534 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdb25\" (UniqueName: \"kubernetes.io/projected/3850aa45-c7d3-4367-883b-177428028837-kube-api-access-vdb25\") pod \"maas-api-revoke-sub-3d4156f1-k7b9c\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:45.659707 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.659629 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3850aa45-c7d3-4367-883b-177428028837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k7b9c\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:45.660223 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.660202 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3850aa45-c7d3-4367-883b-177428028837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k7b9c\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:45.662053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.662032 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3850aa45-c7d3-4367-883b-177428028837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k7b9c\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:45.668682 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.668650 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdb25\" (UniqueName: \"kubernetes.io/projected/3850aa45-c7d3-4367-883b-177428028837-kube-api-access-vdb25\") pod \"maas-api-revoke-sub-3d4156f1-k7b9c\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:45.843147 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:45.843007 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:46.119465 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:46.119435 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c"] Nov 02 18:08:46.121769 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:08:46.121741 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3850aa45_c7d3_4367_883b_177428028837.slice/crio-677d573a91c93943d27e5f9ddd6a6f029cfd99e158c83aba5c398444f212d7ee WatchSource:0}: Error finding container 677d573a91c93943d27e5f9ddd6a6f029cfd99e158c83aba5c398444f212d7ee: Status 404 returned error can't find the container with id 677d573a91c93943d27e5f9ddd6a6f029cfd99e158c83aba5c398444f212d7ee Nov 02 18:08:46.302393 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:46.302358 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" event={"ID":"3850aa45-c7d3-4367-883b-177428028837","Type":"ContainerStarted","Data":"677d573a91c93943d27e5f9ddd6a6f029cfd99e158c83aba5c398444f212d7ee"} Nov 02 18:08:47.309836 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:47.309748 4310 generic.go:356] "Generic (PLEG): container finished" podID="3850aa45-c7d3-4367-883b-177428028837" containerID="25cc1beec9c512df134210fe641ecf9e5ca2a36d6f4f9994bd1ac71459382466" exitCode=0 Nov 02 18:08:47.309836 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:47.309793 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" event={"ID":"3850aa45-c7d3-4367-883b-177428028837","Type":"ContainerDied","Data":"25cc1beec9c512df134210fe641ecf9e5ca2a36d6f4f9994bd1ac71459382466"} Nov 02 18:08:48.395000 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:48.394968 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:48.448110 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:48.448066 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:48.484363 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:48.484335 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3850aa45-c7d3-4367-883b-177428028837-kube-api-access-vdb25\" (UniqueName: \"kubernetes.io/projected/3850aa45-c7d3-4367-883b-177428028837-kube-api-access-vdb25\") pod \"3850aa45-c7d3-4367-883b-177428028837\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " Nov 02 18:08:48.484519 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:48.484465 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3850aa45-c7d3-4367-883b-177428028837-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3850aa45-c7d3-4367-883b-177428028837-maas-api-xks-ca\") pod \"3850aa45-c7d3-4367-883b-177428028837\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " Nov 02 18:08:48.484575 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:48.484517 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3850aa45-c7d3-4367-883b-177428028837-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3850aa45-c7d3-4367-883b-177428028837-maas-api-openshift-ca\") pod \"3850aa45-c7d3-4367-883b-177428028837\" (UID: \"3850aa45-c7d3-4367-883b-177428028837\") " Nov 02 18:08:48.484934 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:48.484903 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3850aa45-c7d3-4367-883b-177428028837-maas-api-openshift-ca" pod "3850aa45-c7d3-4367-883b-177428028837" (UID: "3850aa45-c7d3-4367-883b-177428028837"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:48.486561 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:48.486532 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3850aa45-c7d3-4367-883b-177428028837-maas-api-xks-ca" pod "3850aa45-c7d3-4367-883b-177428028837" (UID: "3850aa45-c7d3-4367-883b-177428028837"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:08:48.486664 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:48.486559 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3850aa45-c7d3-4367-883b-177428028837-kube-api-access-vdb25" pod "3850aa45-c7d3-4367-883b-177428028837" (UID: "3850aa45-c7d3-4367-883b-177428028837"). InnerVolumeSpecName "kube-api-access-vdb25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:48.585493 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:48.585414 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdb25\" (UniqueName: \"kubernetes.io/projected/3850aa45-c7d3-4367-883b-177428028837-kube-api-access-vdb25\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:48.585493 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:48.585444 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3850aa45-c7d3-4367-883b-177428028837-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:48.585493 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:48.585454 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3850aa45-c7d3-4367-883b-177428028837-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:49.321042 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.321014 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" Nov 02 18:08:49.321042 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.321015 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c" event={"ID":"3850aa45-c7d3-4367-883b-177428028837","Type":"ContainerDied","Data":"677d573a91c93943d27e5f9ddd6a6f029cfd99e158c83aba5c398444f212d7ee"} Nov 02 18:08:49.321269 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.321064 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="677d573a91c93943d27e5f9ddd6a6f029cfd99e158c83aba5c398444f212d7ee" Nov 02 18:08:49.507372 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.507343 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:08:49.594235 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594143 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-token\" (UniqueName: \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-token\") pod \"f5ad861f-f9e3-4422-b663-917fafe4df16\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " Nov 02 18:08:49.594235 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594191 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-certs\") pod \"f5ad861f-f9e3-4422-b663-917fafe4df16\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " Nov 02 18:08:49.594476 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594242 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ad861f-f9e3-4422-b663-917fafe4df16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad861f-f9e3-4422-b663-917fafe4df16-istiod-ca-cert\") pod \"f5ad861f-f9e3-4422-b663-917fafe4df16\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " Nov 02 18:08:49.594476 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594275 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-credential-socket\") pod \"f5ad861f-f9e3-4422-b663-917fafe4df16\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " Nov 02 18:08:49.594476 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594314 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-kube-api-access-smdt7\" (UniqueName: \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-kube-api-access-smdt7\") pod \"f5ad861f-f9e3-4422-b663-917fafe4df16\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " Nov 02 18:08:49.594476 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594362 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-data\") pod \"f5ad861f-f9e3-4422-b663-917fafe4df16\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " Nov 02 18:08:49.594476 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594392 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-socket\") pod \"f5ad861f-f9e3-4422-b663-917fafe4df16\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " Nov 02 18:08:49.594476 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594428 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-envoy\") pod \"f5ad861f-f9e3-4422-b663-917fafe4df16\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " Nov 02 18:08:49.594801 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594494 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-podinfo\") pod \"f5ad861f-f9e3-4422-b663-917fafe4df16\" (UID: \"f5ad861f-f9e3-4422-b663-917fafe4df16\") " Nov 02 18:08:49.594801 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594527 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-certs" pod "f5ad861f-f9e3-4422-b663-917fafe4df16" (UID: "f5ad861f-f9e3-4422-b663-917fafe4df16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:49.594801 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594690 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-credential-socket" pod "f5ad861f-f9e3-4422-b663-917fafe4df16" (UID: "f5ad861f-f9e3-4422-b663-917fafe4df16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:49.594801 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594722 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ad861f-f9e3-4422-b663-917fafe4df16-istiod-ca-cert" pod "f5ad861f-f9e3-4422-b663-917fafe4df16" (UID: "f5ad861f-f9e3-4422-b663-917fafe4df16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:08:49.594801 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594726 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-socket" pod "f5ad861f-f9e3-4422-b663-917fafe4df16" (UID: "f5ad861f-f9e3-4422-b663-917fafe4df16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:49.595098 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594884 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:49.595098 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594904 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:49.595098 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594917 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad861f-f9e3-4422-b663-917fafe4df16-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:49.595098 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.594931 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:49.595263 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.595104 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-data" pod "f5ad861f-f9e3-4422-b663-917fafe4df16" (UID: "f5ad861f-f9e3-4422-b663-917fafe4df16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:49.596566 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.596537 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-token" pod "f5ad861f-f9e3-4422-b663-917fafe4df16" (UID: "f5ad861f-f9e3-4422-b663-917fafe4df16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:49.596566 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.596544 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-envoy" pod "f5ad861f-f9e3-4422-b663-917fafe4df16" (UID: "f5ad861f-f9e3-4422-b663-917fafe4df16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:08:49.596937 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.596915 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-kube-api-access-smdt7" pod "f5ad861f-f9e3-4422-b663-917fafe4df16" (UID: "f5ad861f-f9e3-4422-b663-917fafe4df16"). InnerVolumeSpecName "kube-api-access-smdt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:08:49.596981 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.596927 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-podinfo" pod "f5ad861f-f9e3-4422-b663-917fafe4df16" (UID: "f5ad861f-f9e3-4422-b663-917fafe4df16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:08:49.696394 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.696363 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smdt7\" (UniqueName: \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-kube-api-access-smdt7\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:49.696394 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.696391 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:49.696394 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.696400 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:49.696623 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.696409 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:49.696623 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:49.696418 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ad861f-f9e3-4422-b663-917fafe4df16-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:08:50.327494 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.327456 4310 generic.go:356] "Generic (PLEG): container finished" podID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerID="878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb" exitCode=0 Nov 02 18:08:50.327670 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.327503 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" event={"ID":"f5ad861f-f9e3-4422-b663-917fafe4df16","Type":"ContainerDied","Data":"878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb"} Nov 02 18:08:50.327670 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.327514 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" Nov 02 18:08:50.327670 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.327553 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h" event={"ID":"f5ad861f-f9e3-4422-b663-917fafe4df16","Type":"ContainerDied","Data":"5eec78f2217b6f05ac63bac67724a70600b8f20fae282d61ebe3614965624c1b"} Nov 02 18:08:50.327670 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.327576 4310 scope.go:122] "RemoveContainer" containerID="878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb" Nov 02 18:08:50.337421 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.337401 4310 scope.go:122] "RemoveContainer" containerID="87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3" Nov 02 18:08:50.345718 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.345693 4310 scope.go:122] "RemoveContainer" containerID="878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb" Nov 02 18:08:50.346000 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:08:50.345981 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb\": container with ID starting with 878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb not found: ID does not exist" containerID="878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb" Nov 02 18:08:50.346064 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.346009 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb"} err="failed to get container status \"878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb\": rpc error: code = NotFound desc = could not find container \"878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb\": container with ID starting with 878ea89665223a1273c9a31d988de377a0920ed4741df8bbe5283a02664475bb not found: ID does not exist" Nov 02 18:08:50.346064 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.346027 4310 scope.go:122] "RemoveContainer" containerID="87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3" Nov 02 18:08:50.346290 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:08:50.346271 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3\": container with ID starting with 87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3 not found: ID does not exist" containerID="87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3" Nov 02 18:08:50.346337 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.346294 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3"} err="failed to get container status \"87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3\": rpc error: code = NotFound desc = could not find container \"87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3\": container with ID starting with 87b10675dabd0a3602c20af73e02f727bb859eda616ad0dcd55b7a053daf7da3 not found: ID does not exist" Nov 02 18:08:50.350037 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.350015 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h"] Nov 02 18:08:50.353918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.353895 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-836cee-openshift-default-ddd78df65-hw75h"] Nov 02 18:08:50.672501 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:50.672406 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" path="/var/lib/kubelet/pods/f5ad861f-f9e3-4422-b663-917fafe4df16/volumes" Nov 02 18:08:51.818311 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:51.818274 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" podUID="1641d288-9f61-49e4-8119-322785ca136c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:08:53.844829 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:53.844794 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:08:58.027707 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:58.027667 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv"] Nov 02 18:08:58.030918 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:58.030892 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-746lv"] Nov 02 18:08:58.677589 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:08:58.677544 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7034824-3dff-401e-8d5b-89816b1040da" path="/var/lib/kubelet/pods/d7034824-3dff-401e-8d5b-89816b1040da/volumes" Nov 02 18:09:01.241036 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.241006 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:09:01.337864 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.337772 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1641d288-9f61-49e4-8119-322785ca136c-kube-api-access-r8w2p\" (UniqueName: \"kubernetes.io/projected/1641d288-9f61-49e4-8119-322785ca136c-kube-api-access-r8w2p\") pod \"1641d288-9f61-49e4-8119-322785ca136c\" (UID: \"1641d288-9f61-49e4-8119-322785ca136c\") " Nov 02 18:09:01.338052 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.337878 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1641d288-9f61-49e4-8119-322785ca136c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1641d288-9f61-49e4-8119-322785ca136c-plugins-config-volume\") pod \"1641d288-9f61-49e4-8119-322785ca136c\" (UID: \"1641d288-9f61-49e4-8119-322785ca136c\") " Nov 02 18:09:01.338279 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.338254 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1641d288-9f61-49e4-8119-322785ca136c-plugins-config-volume" pod "1641d288-9f61-49e4-8119-322785ca136c" (UID: "1641d288-9f61-49e4-8119-322785ca136c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:09:01.339977 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.339953 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1641d288-9f61-49e4-8119-322785ca136c-kube-api-access-r8w2p" pod "1641d288-9f61-49e4-8119-322785ca136c" (UID: "1641d288-9f61-49e4-8119-322785ca136c"). InnerVolumeSpecName "kube-api-access-r8w2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:09:01.424271 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.424231 4310 generic.go:356] "Generic (PLEG): container finished" podID="1641d288-9f61-49e4-8119-322785ca136c" containerID="9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a" exitCode=0 Nov 02 18:09:01.424477 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.424295 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" event={"ID":"1641d288-9f61-49e4-8119-322785ca136c","Type":"ContainerDied","Data":"9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a"} Nov 02 18:09:01.424477 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.424303 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" Nov 02 18:09:01.424477 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.424365 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4" event={"ID":"1641d288-9f61-49e4-8119-322785ca136c","Type":"ContainerDied","Data":"adfc64eb89b42df067bd71d14f3ab1654577bd120f06a63f5311925fc3c1a904"} Nov 02 18:09:01.424477 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.424387 4310 scope.go:122] "RemoveContainer" containerID="9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a" Nov 02 18:09:01.434583 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.434562 4310 scope.go:122] "RemoveContainer" containerID="9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a" Nov 02 18:09:01.434891 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:09:01.434871 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a\": container with ID starting with 9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a not found: ID does not exist" containerID="9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a" Nov 02 18:09:01.434939 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.434900 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a"} err="failed to get container status \"9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a\": rpc error: code = NotFound desc = could not find container \"9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a\": container with ID starting with 9e0d2f509f4f5e07320b96bd94fafd3dfb626cb2f746163ba552b866714c683a not found: ID does not exist" Nov 02 18:09:01.438495 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.438476 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1641d288-9f61-49e4-8119-322785ca136c-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:01.438550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.438498 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8w2p\" (UniqueName: \"kubernetes.io/projected/1641d288-9f61-49e4-8119-322785ca136c-kube-api-access-r8w2p\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:09:01.447319 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.447297 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4"] Nov 02 18:09:01.450987 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:01.450964 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-836cee-7468dddfx4"] Nov 02 18:09:02.672767 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:02.672727 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1641d288-9f61-49e4-8119-322785ca136c" path="/var/lib/kubelet/pods/1641d288-9f61-49e4-8119-322785ca136c/volumes" Nov 02 18:09:03.523173 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:09:03.523133 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:09:05.431263 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:05.431235 4310 scope.go:122] "RemoveContainer" containerID="6c166379e02691394f717c672fd630bb3c0aa4a97cce75d34ad76c1b1c527dbb" Nov 02 18:09:07.030688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:07.030650 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:07.977731 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:07.977700 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:09.022057 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:09.022018 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds"] Nov 02 18:09:09.025025 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:09.024994 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-wlkds"] Nov 02 18:09:10.672194 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:10.672157 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c8f542-1636-424a-bfcf-8d4aa86e3cdb" path="/var/lib/kubelet/pods/59c8f542-1636-424a-bfcf-8d4aa86e3cdb/volumes" Nov 02 18:09:13.678527 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:13.678480 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:16.430208 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:16.430116 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:16.531659 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:16.531608 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:27.491090 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:27.491031 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:33.522342 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:09:33.522305 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:09:40.042940 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:40.042900 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff"] Nov 02 18:09:40.049934 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:40.049901 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-sf4ff"] Nov 02 18:09:40.671872 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:40.671839 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8951f289-e47f-4258-987f-a92825dbc0ec" path="/var/lib/kubelet/pods/8951f289-e47f-4258-987f-a92825dbc0ec/volumes" Nov 02 18:09:48.442775 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:48.442737 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:09:59.024874 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:59.024829 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q"] Nov 02 18:09:59.026933 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:09:59.026906 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-85t6q"] Nov 02 18:10:00.674619 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:10:00.674575 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459f33a3-087a-471b-a572-e74fab1c8341" path="/var/lib/kubelet/pods/459f33a3-087a-471b-a572-e74fab1c8341/volumes" Nov 02 18:10:03.522694 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:10:03.522659 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:10:05.462169 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:10:05.462122 4310 scope.go:122] "RemoveContainer" containerID="fcad6a36ec9fb3b681e7acc7885f51d136a2fa9676fd4ba8b419f5cc6470b69a" Nov 02 18:10:05.470824 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:10:05.470811 4310 scope.go:122] "RemoveContainer" containerID="df0def6848b4461775c2f1b18ba44e26e31e32b90d177f12653ed86ab36cdd8c" Nov 02 18:10:05.478862 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:10:05.478846 4310 scope.go:122] "RemoveContainer" containerID="7d2620c02ebabec1dfe67e63102b342b6ea27a39ff8f7c06528cf319abbe4eff" Nov 02 18:10:18.021134 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:10:18.021090 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl"] Nov 02 18:10:18.023445 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:10:18.023424 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbtbl"] Nov 02 18:10:18.672505 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:10:18.672464 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7" path="/var/lib/kubelet/pods/b735cfa2-f6a4-4c1d-8aa1-681ecdd3f5f7/volumes" Nov 02 18:10:33.522866 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:10:33.522830 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:10:38.495821 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:10:38.495776 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:10:48.504626 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:10:48.504509 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:03.523102 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:11:03.523034 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:11:05.505435 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:05.505399 4310 scope.go:122] "RemoveContainer" containerID="06634541b1a8621320fb4865e515669497939b77ce0165e5790a927d3b6ea34b" Nov 02 18:11:05.516499 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:05.516478 4310 scope.go:122] "RemoveContainer" containerID="87c06a8741d463645a2243caadd788f8587c026ca84ec5742ef7a9cf86bc66da" Nov 02 18:11:05.524339 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:05.524319 4310 scope.go:122] "RemoveContainer" containerID="f9c6acfc36cb97d4fd2538fd76bce9796df45cf7c0e711157dba9788c584b606" Nov 02 18:11:12.711929 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:12.711890 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:13.087066 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.086981 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf"] Nov 02 18:11:13.087429 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.087414 4310 cpu_manager.go:414] "containerMap: removing container" podUID="1641d288-9f61-49e4-8119-322785ca136c" containerName="payload-pre-processing" Nov 02 18:11:13.087474 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.087431 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1641d288-9f61-49e4-8119-322785ca136c" containerName="payload-pre-processing" Nov 02 18:11:13.087474 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.087443 4310 cpu_manager.go:414] "containerMap: removing container" podUID="3850aa45-c7d3-4367-883b-177428028837" containerName="revoke-keys" Nov 02 18:11:13.087474 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.087449 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3850aa45-c7d3-4367-883b-177428028837" containerName="revoke-keys" Nov 02 18:11:13.087474 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.087458 4310 cpu_manager.go:414] "containerMap: removing container" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" Nov 02 18:11:13.087474 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.087463 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" Nov 02 18:11:13.087638 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.087546 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" Nov 02 18:11:13.087638 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.087558 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="1641d288-9f61-49e4-8119-322785ca136c" containerName="payload-pre-processing" Nov 02 18:11:13.087638 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.087564 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="3850aa45-c7d3-4367-883b-177428028837" containerName="revoke-keys" Nov 02 18:11:13.092328 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.092301 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.100334 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.100309 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-e03252-openshift-default-dockercfg-4skhj\"" Nov 02 18:11:13.103574 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.103550 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf"] Nov 02 18:11:13.140531 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.140505 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:13.170728 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.170697 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-certs\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.170893 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.170736 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npqnv\" (UniqueName: \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-kube-api-access-npqnv\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.170893 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.170758 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66979aed-0079-4af4-9091-6a8397df7ec8-istiod-ca-cert\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.170893 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.170776 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-envoy\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.171021 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.170895 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-credential-socket\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.171021 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.170957 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-socket\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.171021 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.170998 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66979aed-0079-4af4-9091-6a8397df7ec8-istio-podinfo\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.171183 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.171030 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-istio-token\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.171183 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.171130 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-data\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.271937 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.271896 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-credential-socket\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.271955 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-socket\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.271983 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66979aed-0079-4af4-9091-6a8397df7ec8-istio-podinfo\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.272010 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-istio-token\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.272029 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-data\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.272065 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-certs\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.272118 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npqnv\" (UniqueName: \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-kube-api-access-npqnv\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.272146 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66979aed-0079-4af4-9091-6a8397df7ec8-istiod-ca-cert\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.272177 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-envoy\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272664 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.272436 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-credential-socket\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272664 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.272515 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-data\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272664 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.272613 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-socket\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272664 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.272635 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-certs\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.272920 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.272901 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66979aed-0079-4af4-9091-6a8397df7ec8-istiod-ca-cert\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.274427 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.274404 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-envoy\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.274697 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.274680 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66979aed-0079-4af4-9091-6a8397df7ec8-istio-podinfo\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.276331 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.276312 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:13.279973 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.279950 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-istio-token\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.280840 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.280813 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npqnv\" (UniqueName: \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-kube-api-access-npqnv\") pod \"e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.408830 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.408746 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:13.689269 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.689239 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf"] Nov 02 18:11:13.691308 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:11:13.691271 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66979aed_0079_4af4_9091_6a8397df7ec8.slice/crio-35230159cb6dd1cef291aff20a5b93baab71d426ece2ebba15db4d43f2ce8ddd WatchSource:0}: Error finding container 35230159cb6dd1cef291aff20a5b93baab71d426ece2ebba15db4d43f2ce8ddd: Status 404 returned error can't find the container with id 35230159cb6dd1cef291aff20a5b93baab71d426ece2ebba15db4d43f2ce8ddd Nov 02 18:11:13.693828 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.693799 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:11:13.693905 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.693875 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:11:13.693948 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:13.693919 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:11:14.104696 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:14.104654 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" event={"ID":"66979aed-0079-4af4-9091-6a8397df7ec8","Type":"ContainerStarted","Data":"5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419"} Nov 02 18:11:14.105146 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:14.104755 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" event={"ID":"66979aed-0079-4af4-9091-6a8397df7ec8","Type":"ContainerStarted","Data":"35230159cb6dd1cef291aff20a5b93baab71d426ece2ebba15db4d43f2ce8ddd"} Nov 02 18:11:14.129298 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:14.129239 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" podStartSLOduration=1.129219484 podStartE2EDuration="1.129219484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:14.12748261 +0000 UTC m=+1871.570565248" watchObservedRunningTime="2026-11-02 18:11:14.129219484 +0000 UTC m=+1871.572302122" Nov 02 18:11:14.409415 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:14.409321 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:14.414704 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:14.414676 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:15.109053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:15.109020 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:15.109947 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:15.109929 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:16.474294 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:16.474254 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:16.572739 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:16.572708 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:17.484651 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:17.484620 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:17.585322 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:17.585280 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:17.684663 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:17.684635 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:18.272976 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:18.272933 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:18.951450 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:18.951402 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:19.574824 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.574775 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:19.651596 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.651554 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg"] Nov 02 18:11:19.652046 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.652028 4310 cpu_manager.go:414] "containerMap: removing container" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" Nov 02 18:11:19.652169 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.652048 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" Nov 02 18:11:19.652231 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.652210 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ad861f-f9e3-4422-b663-917fafe4df16" containerName="istio-proxy" Nov 02 18:11:19.655178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.655157 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:11:19.661141 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.661115 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e03252\"" Nov 02 18:11:19.661860 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.661839 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5b6769ba-dockercfg-hbdr4\"" Nov 02 18:11:19.664667 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.664643 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg"] Nov 02 18:11:19.676668 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.673989 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:19.742741 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.742704 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg\" (UID: \"d5f85ea1-88f9-4174-a702-93d5cedf8cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:11:19.742945 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.742760 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff8qf\" (UniqueName: \"kubernetes.io/projected/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-kube-api-access-ff8qf\") pod \"payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg\" (UID: \"d5f85ea1-88f9-4174-a702-93d5cedf8cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:11:19.845200 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.844131 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg\" (UID: \"d5f85ea1-88f9-4174-a702-93d5cedf8cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:11:19.846390 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.846366 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff8qf\" (UniqueName: \"kubernetes.io/projected/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-kube-api-access-ff8qf\") pod \"payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg\" (UID: \"d5f85ea1-88f9-4174-a702-93d5cedf8cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:11:19.846897 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.846303 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg\" (UID: \"d5f85ea1-88f9-4174-a702-93d5cedf8cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:11:19.858620 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.858588 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff8qf\" (UniqueName: \"kubernetes.io/projected/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-kube-api-access-ff8qf\") pod \"payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg\" (UID: \"d5f85ea1-88f9-4174-a702-93d5cedf8cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:11:19.970581 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:19.970542 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:11:20.260125 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:20.260101 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg"] Nov 02 18:11:20.262294 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:11:20.262256 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f85ea1_88f9_4174_a702_93d5cedf8cfa.slice/crio-088516bb0b87010cc62c4cac787f11a999004444d4749066ef87a554cf186717 WatchSource:0}: Error finding container 088516bb0b87010cc62c4cac787f11a999004444d4749066ef87a554cf186717: Status 404 returned error can't find the container with id 088516bb0b87010cc62c4cac787f11a999004444d4749066ef87a554cf186717 Nov 02 18:11:20.449996 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:20.449963 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:21.137012 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:21.136973 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" event={"ID":"d5f85ea1-88f9-4174-a702-93d5cedf8cfa","Type":"ContainerStarted","Data":"b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a"} Nov 02 18:11:21.137549 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:21.137036 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" event={"ID":"d5f85ea1-88f9-4174-a702-93d5cedf8cfa","Type":"ContainerStarted","Data":"088516bb0b87010cc62c4cac787f11a999004444d4749066ef87a554cf186717"} Nov 02 18:11:21.137549 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:21.137111 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:11:21.185786 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:21.185753 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:21.977737 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:21.977701 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:22.672924 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:22.672876 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:23.487038 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:23.486990 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:24.280307 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:24.280264 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:24.890313 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:24.890279 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:25.571407 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:25.571363 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:26.252558 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:26.252526 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:26.936398 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:26.936361 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:27.664825 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:27.664786 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:28.337473 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:28.337432 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:28.945954 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:28.945911 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:29.669743 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:29.669709 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:30.110145 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:30.110112 4310 patch_prober.go:29] interesting pod/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 18:11:30.110326 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:30.110168 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 18:11:30.206221 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:30.206181 4310 generic.go:356] "Generic (PLEG): container finished" podID="66979aed-0079-4af4-9091-6a8397df7ec8" containerID="5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419" exitCode=0 Nov 02 18:11:30.206397 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:30.206232 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" event={"ID":"66979aed-0079-4af4-9091-6a8397df7ec8","Type":"ContainerDied","Data":"5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419"} Nov 02 18:11:30.206614 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:30.206600 4310 scope.go:122] "RemoveContainer" containerID="5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419" Nov 02 18:11:30.207851 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:30.207823 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:11:30.207912 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:30.207895 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:11:30.207948 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:30.207924 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:11:30.231757 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:30.231701 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" podStartSLOduration=11.231684697 podStartE2EDuration="11.231684697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:21.156344984 +0000 UTC m=+1878.599427623" watchObservedRunningTime="2026-11-02 18:11:30.231684697 +0000 UTC m=+1887.674767332" Nov 02 18:11:30.347972 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:30.345568 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:30.988567 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:30.988527 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:31.213844 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:31.213808 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" event={"ID":"66979aed-0079-4af4-9091-6a8397df7ec8","Type":"ContainerStarted","Data":"7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83"} Nov 02 18:11:31.409877 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:31.409781 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:31.410067 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:31.409994 4310 patch_prober.go:29] interesting pod/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 18:11:31.410184 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:31.410065 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 18:11:31.680777 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:31.680690 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:32.168022 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:32.167991 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:11:32.396446 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:32.396409 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:32.409583 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:32.409550 4310 patch_prober.go:29] interesting pod/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 18:11:32.409747 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:32.409614 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 18:11:33.120356 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:33.120318 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:33.409202 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:33.409119 4310 patch_prober.go:29] interesting pod/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 18:11:33.409202 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:33.409173 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 18:11:33.522950 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:11:33.522911 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:11:33.792160 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:33.792123 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:34.410031 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:34.409997 4310 patch_prober.go:29] interesting pod/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 18:11:34.410478 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:34.410060 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 18:11:34.480507 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:34.480471 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:35.168805 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:35.168762 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:35.410029 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:35.409993 4310 patch_prober.go:29] interesting pod/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 18:11:35.410216 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:35.410056 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 18:11:35.868602 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:35.868566 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:36.413675 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:36.413641 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:36.414057 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:36.413913 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:36.414564 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:36.414548 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:11:36.489161 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:36.489122 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:37.156704 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:37.156673 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:37.839775 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:37.839740 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:38.228096 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:38.228026 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:38.429343 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:38.429306 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:38.846831 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:38.846793 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:38.947005 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:38.946954 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:39.084799 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:39.084753 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:39.252487 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:39.252446 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:40.080628 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:40.080590 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:40.180337 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:40.180295 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:40.765359 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:40.765313 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:41.441015 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:41.440979 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:42.265417 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:42.265379 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:42.979114 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:42.979063 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:43.749742 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:43.749706 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:44.555428 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:44.555379 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:45.176786 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:45.176738 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:45.863330 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:45.863287 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:46.569899 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:46.569861 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:47.355545 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:47.355510 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:47.996792 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:47.996754 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:48.526042 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:48.526000 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:48.660694 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:48.660649 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:49.306440 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:49.306397 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:50.080810 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:50.080775 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:50.666414 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:50.666370 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:51.379326 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:51.379292 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:52.163695 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:52.163664 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:52.764674 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:52.764637 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:53.471490 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:53.471446 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:54.071603 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:54.071564 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:54.764585 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:54.764550 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:55.545470 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:55.545431 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:56.215969 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:56.215933 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:56.947100 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:56.947040 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:57.577986 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:57.577936 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:58.370747 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:58.370692 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:59.178756 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:59.178719 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:11:59.845330 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:11:59.845302 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:00.479832 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:00.479789 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:01.184987 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:01.184953 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:01.823762 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:01.823721 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:02.463547 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:02.463505 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:03.191203 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:03.191166 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:03.522241 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:12:03.522204 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:12:03.965339 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:03.965292 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:04.561095 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:04.561045 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:05.170409 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:05.170369 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:05.550258 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:05.550225 4310 scope.go:122] "RemoveContainer" containerID="41fa412b7c2c2979efcdc12e4efd77eb2f0a5d0ff0a703096c3500f939a92da9" Nov 02 18:12:05.772516 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:05.772482 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:06.455256 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:06.455218 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:07.067043 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:07.067006 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:07.795534 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:07.795501 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:08.321829 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:08.321795 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:08.422866 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:08.422827 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:08.986329 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:08.986291 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:09.568301 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:09.568268 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:10.380406 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.380375 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:10.412382 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.412345 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj"] Nov 02 18:12:10.415444 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.415426 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:10.426060 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.426023 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj"] Nov 02 18:12:10.426572 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.426542 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:12:10.427619 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.426803 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:12:10.497786 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.497751 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nqlq\" (UniqueName: \"kubernetes.io/projected/c5baccd4-9912-470d-af3e-ada2b40c3e20-kube-api-access-5nqlq\") pod \"maas-api-revoke-sub-87ce9c90-nx5tj\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:10.497949 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.497816 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87ce9c90-nx5tj\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:10.497949 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.497909 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87ce9c90-nx5tj\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:10.598897 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.598865 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87ce9c90-nx5tj\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:10.599374 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.598925 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87ce9c90-nx5tj\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:10.599374 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.598981 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nqlq\" (UniqueName: \"kubernetes.io/projected/c5baccd4-9912-470d-af3e-ada2b40c3e20-kube-api-access-5nqlq\") pod \"maas-api-revoke-sub-87ce9c90-nx5tj\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:10.599664 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.599641 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87ce9c90-nx5tj\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:10.601199 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.601179 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87ce9c90-nx5tj\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:10.612487 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.612463 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nqlq\" (UniqueName: \"kubernetes.io/projected/c5baccd4-9912-470d-af3e-ada2b40c3e20-kube-api-access-5nqlq\") pod \"maas-api-revoke-sub-87ce9c90-nx5tj\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:10.758151 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:10.758117 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:11.030588 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:11.030562 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj"] Nov 02 18:12:11.032858 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:12:11.032810 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5baccd4_9912_470d_af3e_ada2b40c3e20.slice/crio-03ded08ee48a67d1d98d255fa5c46b7da2ed979a8fb1ddc558d048bc362d01b7 WatchSource:0}: Error finding container 03ded08ee48a67d1d98d255fa5c46b7da2ed979a8fb1ddc558d048bc362d01b7: Status 404 returned error can't find the container with id 03ded08ee48a67d1d98d255fa5c46b7da2ed979a8fb1ddc558d048bc362d01b7 Nov 02 18:12:11.164420 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:11.164382 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:11.466477 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:11.466438 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" event={"ID":"c5baccd4-9912-470d-af3e-ada2b40c3e20","Type":"ContainerStarted","Data":"0c34bec27ef9fcc5e0c806f2c279e37251c23a3b07be15a3da01ea173993a270"} Nov 02 18:12:11.466704 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:11.466500 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" event={"ID":"c5baccd4-9912-470d-af3e-ada2b40c3e20","Type":"ContainerStarted","Data":"03ded08ee48a67d1d98d255fa5c46b7da2ed979a8fb1ddc558d048bc362d01b7"} Nov 02 18:12:11.494522 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:11.494473 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" podStartSLOduration=1.494458254 podStartE2EDuration="1.494458254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:11.492360176 +0000 UTC m=+1928.935442813" watchObservedRunningTime="2026-11-02 18:12:11.494458254 +0000 UTC m=+1928.937540928" Nov 02 18:12:11.969654 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:11.969618 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:12.473315 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:12.473278 4310 generic.go:356] "Generic (PLEG): container finished" podID="c5baccd4-9912-470d-af3e-ada2b40c3e20" containerID="0c34bec27ef9fcc5e0c806f2c279e37251c23a3b07be15a3da01ea173993a270" exitCode=0 Nov 02 18:12:12.473504 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:12.473366 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" event={"ID":"c5baccd4-9912-470d-af3e-ada2b40c3e20","Type":"ContainerDied","Data":"0c34bec27ef9fcc5e0c806f2c279e37251c23a3b07be15a3da01ea173993a270"} Nov 02 18:12:12.568503 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:12.568471 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:13.167630 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.167600 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:13.616104 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.616054 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:13.632041 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.632004 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-openshift-ca\") pod \"c5baccd4-9912-470d-af3e-ada2b40c3e20\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " Nov 02 18:12:13.632264 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.632064 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5baccd4-9912-470d-af3e-ada2b40c3e20-kube-api-access-5nqlq\" (UniqueName: \"kubernetes.io/projected/c5baccd4-9912-470d-af3e-ada2b40c3e20-kube-api-access-5nqlq\") pod \"c5baccd4-9912-470d-af3e-ada2b40c3e20\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " Nov 02 18:12:13.632264 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.632237 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-xks-ca\") pod \"c5baccd4-9912-470d-af3e-ada2b40c3e20\" (UID: \"c5baccd4-9912-470d-af3e-ada2b40c3e20\") " Nov 02 18:12:13.633226 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.633191 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-openshift-ca" pod "c5baccd4-9912-470d-af3e-ada2b40c3e20" (UID: "c5baccd4-9912-470d-af3e-ada2b40c3e20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:13.635105 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.635060 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-xks-ca" pod "c5baccd4-9912-470d-af3e-ada2b40c3e20" (UID: "c5baccd4-9912-470d-af3e-ada2b40c3e20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:13.635580 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.635558 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5baccd4-9912-470d-af3e-ada2b40c3e20-kube-api-access-5nqlq" pod "c5baccd4-9912-470d-af3e-ada2b40c3e20" (UID: "c5baccd4-9912-470d-af3e-ada2b40c3e20"). InnerVolumeSpecName "kube-api-access-5nqlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:13.733683 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.733583 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:13.733683 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.733629 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5baccd4-9912-470d-af3e-ada2b40c3e20-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:13.733683 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.733641 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nqlq\" (UniqueName: \"kubernetes.io/projected/c5baccd4-9912-470d-af3e-ada2b40c3e20-kube-api-access-5nqlq\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:13.854020 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:13.853984 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:14.457105 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:14.457047 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:14.484510 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:14.484482 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" Nov 02 18:12:14.484680 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:14.484517 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj" event={"ID":"c5baccd4-9912-470d-af3e-ada2b40c3e20","Type":"ContainerDied","Data":"03ded08ee48a67d1d98d255fa5c46b7da2ed979a8fb1ddc558d048bc362d01b7"} Nov 02 18:12:14.484680 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:14.484568 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03ded08ee48a67d1d98d255fa5c46b7da2ed979a8fb1ddc558d048bc362d01b7" Nov 02 18:12:15.104768 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:15.104734 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:15.674368 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:15.674333 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:16.259489 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:16.259455 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:16.893875 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:16.893835 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:17.567055 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:17.567011 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:18.247723 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:18.247685 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:18.787547 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:18.787511 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:19.495590 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:19.495559 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:20.213280 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:20.213233 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:20.785129 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:20.785096 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:21.225279 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.225249 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:21.482608 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.482523 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l"] Nov 02 18:12:21.482974 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.482960 4310 cpu_manager.go:414] "containerMap: removing container" podUID="c5baccd4-9912-470d-af3e-ada2b40c3e20" containerName="revoke-keys" Nov 02 18:12:21.483019 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.482975 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5baccd4-9912-470d-af3e-ada2b40c3e20" containerName="revoke-keys" Nov 02 18:12:21.483092 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.483070 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5baccd4-9912-470d-af3e-ada2b40c3e20" containerName="revoke-keys" Nov 02 18:12:21.485330 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.485307 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:21.490022 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.489999 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:12:21.492515 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.492496 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:12:21.498511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.498487 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l"] Nov 02 18:12:21.510668 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.510638 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:21.510811 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.510725 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bxdp\" (UniqueName: \"kubernetes.io/projected/56a5d781-67ac-4cef-99af-a6a73b1e3934-kube-api-access-2bxdp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:21.510811 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.510765 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:21.565836 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.565790 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:21.612239 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.612194 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:21.612429 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.612276 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxdp\" (UniqueName: \"kubernetes.io/projected/56a5d781-67ac-4cef-99af-a6a73b1e3934-kube-api-access-2bxdp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:21.612429 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.612303 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:21.612983 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.612959 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:21.614598 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.614578 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:21.620898 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.620876 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bxdp\" (UniqueName: \"kubernetes.io/projected/56a5d781-67ac-4cef-99af-a6a73b1e3934-kube-api-access-2bxdp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:21.799789 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:21.799705 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:22.075233 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:22.075209 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l"] Nov 02 18:12:22.077343 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:12:22.077316 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a5d781_67ac_4cef_99af_a6a73b1e3934.slice/crio-5696fabc0d961e913b304c712754e7d45f99ca55aa69d19fed6e00c69726facf WatchSource:0}: Error finding container 5696fabc0d961e913b304c712754e7d45f99ca55aa69d19fed6e00c69726facf: Status 404 returned error can't find the container with id 5696fabc0d961e913b304c712754e7d45f99ca55aa69d19fed6e00c69726facf Nov 02 18:12:22.363710 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:22.363682 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:22.524603 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:22.524562 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" event={"ID":"56a5d781-67ac-4cef-99af-a6a73b1e3934","Type":"ContainerStarted","Data":"b75c44e44e547f7daa082c39b809cd7fa2d9eaa0675893a1f7394ac4239e3a60"} Nov 02 18:12:22.524815 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:22.524615 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" event={"ID":"56a5d781-67ac-4cef-99af-a6a73b1e3934","Type":"ContainerStarted","Data":"5696fabc0d961e913b304c712754e7d45f99ca55aa69d19fed6e00c69726facf"} Nov 02 18:12:22.546704 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:22.544917 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" podStartSLOduration=1.544898435 podStartE2EDuration="1.544898435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:22.540188399 +0000 UTC m=+1939.983271035" watchObservedRunningTime="2026-11-02 18:12:22.544898435 +0000 UTC m=+1939.987981071" Nov 02 18:12:23.071782 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:23.071743 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:23.530970 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:23.530935 4310 generic.go:356] "Generic (PLEG): container finished" podID="56a5d781-67ac-4cef-99af-a6a73b1e3934" containerID="b75c44e44e547f7daa082c39b809cd7fa2d9eaa0675893a1f7394ac4239e3a60" exitCode=0 Nov 02 18:12:23.531179 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:23.531033 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" event={"ID":"56a5d781-67ac-4cef-99af-a6a73b1e3934","Type":"ContainerDied","Data":"b75c44e44e547f7daa082c39b809cd7fa2d9eaa0675893a1f7394ac4239e3a60"} Nov 02 18:12:23.754553 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:23.754520 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:24.372364 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:24.372327 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:24.673031 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:24.673003 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:24.749219 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:24.749180 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a5d781-67ac-4cef-99af-a6a73b1e3934-kube-api-access-2bxdp\" (UniqueName: \"kubernetes.io/projected/56a5d781-67ac-4cef-99af-a6a73b1e3934-kube-api-access-2bxdp\") pod \"56a5d781-67ac-4cef-99af-a6a73b1e3934\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " Nov 02 18:12:24.749421 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:24.749278 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-openshift-ca\") pod \"56a5d781-67ac-4cef-99af-a6a73b1e3934\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " Nov 02 18:12:24.749421 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:24.749364 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-xks-ca\") pod \"56a5d781-67ac-4cef-99af-a6a73b1e3934\" (UID: \"56a5d781-67ac-4cef-99af-a6a73b1e3934\") " Nov 02 18:12:24.749666 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:24.749643 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-openshift-ca" pod "56a5d781-67ac-4cef-99af-a6a73b1e3934" (UID: "56a5d781-67ac-4cef-99af-a6a73b1e3934"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:24.749878 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:24.749858 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:24.751461 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:24.751426 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a5d781-67ac-4cef-99af-a6a73b1e3934-kube-api-access-2bxdp" pod "56a5d781-67ac-4cef-99af-a6a73b1e3934" (UID: "56a5d781-67ac-4cef-99af-a6a73b1e3934"). InnerVolumeSpecName "kube-api-access-2bxdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:24.751564 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:24.751544 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-xks-ca" pod "56a5d781-67ac-4cef-99af-a6a73b1e3934" (UID: "56a5d781-67ac-4cef-99af-a6a73b1e3934"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:12:24.851245 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:24.851209 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bxdp\" (UniqueName: \"kubernetes.io/projected/56a5d781-67ac-4cef-99af-a6a73b1e3934-kube-api-access-2bxdp\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:24.851245 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:24.851239 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5d781-67ac-4cef-99af-a6a73b1e3934-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:25.167162 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:25.167128 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:25.542437 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:25.542392 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" event={"ID":"56a5d781-67ac-4cef-99af-a6a73b1e3934","Type":"ContainerDied","Data":"5696fabc0d961e913b304c712754e7d45f99ca55aa69d19fed6e00c69726facf"} Nov 02 18:12:25.542437 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:25.542413 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l" Nov 02 18:12:25.542919 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:25.542449 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5696fabc0d961e913b304c712754e7d45f99ca55aa69d19fed6e00c69726facf" Nov 02 18:12:25.856494 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:25.856413 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:26.292597 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:26.292519 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:26.563703 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:26.563576 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:27.267867 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:27.267827 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:27.930616 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:27.930573 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:28.615401 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:28.615363 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:29.266625 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:29.266589 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:29.870859 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:29.870822 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:30.551050 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:30.551008 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:31.162969 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:31.162898 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:31.610071 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:31.610030 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg"] Nov 02 18:12:31.676841 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:31.676801 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:31.772026 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:31.771988 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:31.989469 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:31.989427 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:32.000504 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:32.000471 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l"] Nov 02 18:12:32.005633 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:32.005601 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e03252-gpd2l"] Nov 02 18:12:32.089270 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:32.089180 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:32.501298 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:32.501259 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:32.672251 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:32.672211 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a5d781-67ac-4cef-99af-a6a73b1e3934" path="/var/lib/kubelet/pods/56a5d781-67ac-4cef-99af-a6a73b1e3934/volumes" Nov 02 18:12:33.522615 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:12:33.522577 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:12:36.611200 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:36.611140 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" podUID="d5f85ea1-88f9-4174-a702-93d5cedf8cfa" containerName="payload-pre-processing" containerID="cri-o://b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a" gracePeriod=25 Nov 02 18:12:36.857272 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:36.857245 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:12:36.999345 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:36.999311 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-kube-api-access-ff8qf\" (UniqueName: \"kubernetes.io/projected/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-kube-api-access-ff8qf\") pod \"d5f85ea1-88f9-4174-a702-93d5cedf8cfa\" (UID: \"d5f85ea1-88f9-4174-a702-93d5cedf8cfa\") " Nov 02 18:12:36.999543 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:36.999482 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-plugins-config-volume\") pod \"d5f85ea1-88f9-4174-a702-93d5cedf8cfa\" (UID: \"d5f85ea1-88f9-4174-a702-93d5cedf8cfa\") " Nov 02 18:12:36.999828 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:36.999806 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-plugins-config-volume" pod "d5f85ea1-88f9-4174-a702-93d5cedf8cfa" (UID: "d5f85ea1-88f9-4174-a702-93d5cedf8cfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:37.001402 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.001374 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-kube-api-access-ff8qf" pod "d5f85ea1-88f9-4174-a702-93d5cedf8cfa" (UID: "d5f85ea1-88f9-4174-a702-93d5cedf8cfa"). InnerVolumeSpecName "kube-api-access-ff8qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:37.100589 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.100548 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:37.100589 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.100578 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff8qf\" (UniqueName: \"kubernetes.io/projected/d5f85ea1-88f9-4174-a702-93d5cedf8cfa-kube-api-access-ff8qf\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:37.608026 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.607974 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:37.615117 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.615067 4310 generic.go:356] "Generic (PLEG): container finished" podID="d5f85ea1-88f9-4174-a702-93d5cedf8cfa" containerID="b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a" exitCode=0 Nov 02 18:12:37.615117 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.615107 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" event={"ID":"d5f85ea1-88f9-4174-a702-93d5cedf8cfa","Type":"ContainerDied","Data":"b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a"} Nov 02 18:12:37.615570 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.615141 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" Nov 02 18:12:37.615570 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.615165 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg" event={"ID":"d5f85ea1-88f9-4174-a702-93d5cedf8cfa","Type":"ContainerDied","Data":"088516bb0b87010cc62c4cac787f11a999004444d4749066ef87a554cf186717"} Nov 02 18:12:37.615570 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.615188 4310 scope.go:122] "RemoveContainer" containerID="b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a" Nov 02 18:12:37.624150 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.624127 4310 scope.go:122] "RemoveContainer" containerID="b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a" Nov 02 18:12:37.624420 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:12:37.624398 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a\": container with ID starting with b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a not found: ID does not exist" containerID="b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a" Nov 02 18:12:37.624465 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.624430 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a"} err="failed to get container status \"b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a\": rpc error: code = NotFound desc = could not find container \"b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a\": container with ID starting with b2b11e129b8580c5d53d4a2e3c70287add220eedb6b181ace4cc9ad2b05cdd2a not found: ID does not exist" Nov 02 18:12:37.637002 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.636965 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg"] Nov 02 18:12:37.641397 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:37.641374 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e03252-67697xp8cg"] Nov 02 18:12:38.672459 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:38.672428 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f85ea1-88f9-4174-a702-93d5cedf8cfa" path="/var/lib/kubelet/pods/d5f85ea1-88f9-4174-a702-93d5cedf8cfa/volumes" Nov 02 18:12:38.823313 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:38.823277 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:38.934521 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:38.934443 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:39.090947 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:39.090902 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf"] Nov 02 18:12:39.091299 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:39.091264 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" containerID="cri-o://7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83" gracePeriod=30 Nov 02 18:12:39.130281 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:39.130192 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:43.814743 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:43.814704 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:44.343722 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.343695 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:12:44.373712 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.373638 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-data\") pod \"66979aed-0079-4af4-9091-6a8397df7ec8\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " Nov 02 18:12:44.373712 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.373683 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-kube-api-access-npqnv\" (UniqueName: \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-kube-api-access-npqnv\") pod \"66979aed-0079-4af4-9091-6a8397df7ec8\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " Nov 02 18:12:44.373942 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.373715 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-istio-token\" (UniqueName: \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-istio-token\") pod \"66979aed-0079-4af4-9091-6a8397df7ec8\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " Nov 02 18:12:44.373942 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.373754 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66979aed-0079-4af4-9091-6a8397df7ec8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66979aed-0079-4af4-9091-6a8397df7ec8-istiod-ca-cert\") pod \"66979aed-0079-4af4-9091-6a8397df7ec8\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " Nov 02 18:12:44.373942 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.373803 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-envoy\") pod \"66979aed-0079-4af4-9091-6a8397df7ec8\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " Nov 02 18:12:44.373942 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.373853 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-certs\") pod \"66979aed-0079-4af4-9091-6a8397df7ec8\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " Nov 02 18:12:44.373942 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.373886 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-socket\") pod \"66979aed-0079-4af4-9091-6a8397df7ec8\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " Nov 02 18:12:44.373942 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.373937 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-credential-socket\") pod \"66979aed-0079-4af4-9091-6a8397df7ec8\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " Nov 02 18:12:44.374304 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.373963 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-data" pod "66979aed-0079-4af4-9091-6a8397df7ec8" (UID: "66979aed-0079-4af4-9091-6a8397df7ec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:44.374304 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.373993 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66979aed-0079-4af4-9091-6a8397df7ec8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66979aed-0079-4af4-9091-6a8397df7ec8-istio-podinfo\") pod \"66979aed-0079-4af4-9091-6a8397df7ec8\" (UID: \"66979aed-0079-4af4-9091-6a8397df7ec8\") " Nov 02 18:12:44.374304 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.374180 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-certs" pod "66979aed-0079-4af4-9091-6a8397df7ec8" (UID: "66979aed-0079-4af4-9091-6a8397df7ec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:44.374304 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.374210 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66979aed-0079-4af4-9091-6a8397df7ec8-istiod-ca-cert" pod "66979aed-0079-4af4-9091-6a8397df7ec8" (UID: "66979aed-0079-4af4-9091-6a8397df7ec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:44.374521 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.374400 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-socket" pod "66979aed-0079-4af4-9091-6a8397df7ec8" (UID: "66979aed-0079-4af4-9091-6a8397df7ec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:44.374581 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.374534 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:44.374638 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.374580 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66979aed-0079-4af4-9091-6a8397df7ec8-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:44.374638 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.374590 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-credential-socket" pod "66979aed-0079-4af4-9091-6a8397df7ec8" (UID: "66979aed-0079-4af4-9091-6a8397df7ec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:44.374638 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.374597 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:44.374638 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.374635 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:44.376290 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.376267 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-envoy" pod "66979aed-0079-4af4-9091-6a8397df7ec8" (UID: "66979aed-0079-4af4-9091-6a8397df7ec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:44.376290 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.376278 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-kube-api-access-npqnv" pod "66979aed-0079-4af4-9091-6a8397df7ec8" (UID: "66979aed-0079-4af4-9091-6a8397df7ec8"). InnerVolumeSpecName "kube-api-access-npqnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:44.376471 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.376453 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66979aed-0079-4af4-9091-6a8397df7ec8-istio-podinfo" pod "66979aed-0079-4af4-9091-6a8397df7ec8" (UID: "66979aed-0079-4af4-9091-6a8397df7ec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:44.376548 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.376533 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-istio-token" pod "66979aed-0079-4af4-9091-6a8397df7ec8" (UID: "66979aed-0079-4af4-9091-6a8397df7ec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:44.475550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.475508 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:44.475550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.475545 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66979aed-0079-4af4-9091-6a8397df7ec8-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:44.475550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.475555 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66979aed-0079-4af4-9091-6a8397df7ec8-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:44.475804 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.475564 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npqnv\" (UniqueName: \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-kube-api-access-npqnv\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:44.475804 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.475573 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66979aed-0079-4af4-9091-6a8397df7ec8-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:12:44.650973 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.650880 4310 generic.go:356] "Generic (PLEG): container finished" podID="66979aed-0079-4af4-9091-6a8397df7ec8" containerID="7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83" exitCode=0 Nov 02 18:12:44.650973 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.650917 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" event={"ID":"66979aed-0079-4af4-9091-6a8397df7ec8","Type":"ContainerDied","Data":"7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83"} Nov 02 18:12:44.651262 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.650994 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" event={"ID":"66979aed-0079-4af4-9091-6a8397df7ec8","Type":"ContainerDied","Data":"35230159cb6dd1cef291aff20a5b93baab71d426ece2ebba15db4d43f2ce8ddd"} Nov 02 18:12:44.651262 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.650933 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf" Nov 02 18:12:44.651262 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.651026 4310 scope.go:122] "RemoveContainer" containerID="7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83" Nov 02 18:12:44.660340 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.660317 4310 scope.go:122] "RemoveContainer" containerID="5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419" Nov 02 18:12:44.668752 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.668735 4310 scope.go:122] "RemoveContainer" containerID="7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83" Nov 02 18:12:44.669007 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:12:44.668985 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83\": container with ID starting with 7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83 not found: ID does not exist" containerID="7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83" Nov 02 18:12:44.669067 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.669019 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83"} err="failed to get container status \"7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83\": rpc error: code = NotFound desc = could not find container \"7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83\": container with ID starting with 7767752d4b36e8d61a1073763ddf5c62382629ccf2ca1dab5d67ecd2bc07fd83 not found: ID does not exist" Nov 02 18:12:44.669067 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.669037 4310 scope.go:122] "RemoveContainer" containerID="5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419" Nov 02 18:12:44.669281 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:12:44.669264 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419\": container with ID starting with 5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419 not found: ID does not exist" containerID="5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419" Nov 02 18:12:44.669320 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.669285 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419"} err="failed to get container status \"5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419\": rpc error: code = NotFound desc = could not find container \"5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419\": container with ID starting with 5f380ed8375d6ce97d9254776d657d14ca7c9d9deaa5b2134803e27a2b873419 not found: ID does not exist" Nov 02 18:12:44.675301 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.675273 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf"] Nov 02 18:12:44.678267 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:44.678242 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e03252-openshift-default-6cd9f59ff7dqkqf"] Nov 02 18:12:46.680288 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:46.680250 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" path="/var/lib/kubelet/pods/66979aed-0079-4af4-9091-6a8397df7ec8/volumes" Nov 02 18:12:48.719512 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:48.719461 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:59.694162 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.694119 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv"] Nov 02 18:12:59.695484 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695451 4310 cpu_manager.go:414] "containerMap: removing container" podUID="d5f85ea1-88f9-4174-a702-93d5cedf8cfa" containerName="payload-pre-processing" Nov 02 18:12:59.695484 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695481 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f85ea1-88f9-4174-a702-93d5cedf8cfa" containerName="payload-pre-processing" Nov 02 18:12:59.695677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695514 4310 cpu_manager.go:414] "containerMap: removing container" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" Nov 02 18:12:59.695677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695524 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" Nov 02 18:12:59.695677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695536 4310 cpu_manager.go:414] "containerMap: removing container" podUID="56a5d781-67ac-4cef-99af-a6a73b1e3934" containerName="revoke-keys" Nov 02 18:12:59.695677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695545 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a5d781-67ac-4cef-99af-a6a73b1e3934" containerName="revoke-keys" Nov 02 18:12:59.695677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695574 4310 cpu_manager.go:414] "containerMap: removing container" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" Nov 02 18:12:59.695677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695582 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" Nov 02 18:12:59.695677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695650 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a5d781-67ac-4cef-99af-a6a73b1e3934" containerName="revoke-keys" Nov 02 18:12:59.695677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695662 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" Nov 02 18:12:59.695677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695669 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f85ea1-88f9-4174-a702-93d5cedf8cfa" containerName="payload-pre-processing" Nov 02 18:12:59.695677 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.695677 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="66979aed-0079-4af4-9091-6a8397df7ec8" containerName="istio-proxy" Nov 02 18:12:59.698652 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.698634 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.704353 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.704323 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a6fb1955-kserve-self-signed-certs\"" Nov 02 18:12:59.709104 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.708685 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv"] Nov 02 18:12:59.747190 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.747150 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:12:59.753725 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.753699 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-home\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.753867 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.753753 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-model-cache\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.753915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.753856 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-dshm\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.753966 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.753954 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98gp\" (UniqueName: \"kubernetes.io/projected/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-kube-api-access-g98gp\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.754056 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.754044 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tmp-dir\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.754114 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.754069 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tls-certs\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.855483 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.855441 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-model-cache\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.855483 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.855497 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-dshm\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.855782 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.855598 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98gp\" (UniqueName: \"kubernetes.io/projected/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-kube-api-access-g98gp\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.855782 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.855682 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tmp-dir\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.855782 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.855732 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tls-certs\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.855782 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.855766 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-home\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.856435 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.856402 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-home\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.856591 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.856452 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-model-cache\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.856858 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.856835 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tmp-dir\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.859635 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.859611 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tls-certs\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.864697 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.864671 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-dshm\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:12:59.865366 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:12:59.865342 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98gp\" (UniqueName: \"kubernetes.io/projected/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-kube-api-access-g98gp\") pod \"e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:13:00.022003 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:00.021960 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:13:00.295688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:00.295662 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv"] Nov 02 18:13:00.297847 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:13:00.297812 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode821ae21_b5ee_4eac_9bc6_c6de903df2bf.slice/crio-70be2bf70bd314acd64868836547c0fe15c4f12088911710135312a3129235ff WatchSource:0}: Error finding container 70be2bf70bd314acd64868836547c0fe15c4f12088911710135312a3129235ff: Status 404 returned error can't find the container with id 70be2bf70bd314acd64868836547c0fe15c4f12088911710135312a3129235ff Nov 02 18:13:00.761740 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:00.761693 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" event={"ID":"e821ae21-b5ee-4eac-9bc6-c6de903df2bf","Type":"ContainerStarted","Data":"140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513"} Nov 02 18:13:00.761740 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:00.761746 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" event={"ID":"e821ae21-b5ee-4eac-9bc6-c6de903df2bf","Type":"ContainerStarted","Data":"70be2bf70bd314acd64868836547c0fe15c4f12088911710135312a3129235ff"} Nov 02 18:13:00.781683 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:00.781626 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" podStartSLOduration=1.7816092399999999 podStartE2EDuration="1.78160924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:00.779219544 +0000 UTC m=+1978.222302180" watchObservedRunningTime="2026-11-02 18:13:00.78160924 +0000 UTC m=+1978.224691889" Nov 02 18:13:01.022907 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:01.022824 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:13:02.940744 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:02.940697 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:03.537527 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:13:03.537488 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:13:03.585103 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:03.584098 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:09.055113 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:09.055048 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:10.022986 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:10.022951 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:13:10.035734 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:10.035699 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:13:10.849544 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:10.849507 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:13:19.021850 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:19.021811 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b"] Nov 02 18:13:19.024330 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:19.024306 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc483f5f-fjl4b"] Nov 02 18:13:20.716043 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:20.715998 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d948f6-3fb1-4a14-addb-a36c7e95a2f7" path="/var/lib/kubelet/pods/22d948f6-3fb1-4a14-addb-a36c7e95a2f7/volumes" Nov 02 18:13:27.459994 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:27.459931 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:28.939253 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:28.939218 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv"] Nov 02 18:13:28.980769 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:28.980739 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:33.527641 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:13:33.527594 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:13:43.961606 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:43.961541 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" podUID="e821ae21-b5ee-4eac-9bc6-c6de903df2bf" containerName="main" containerID="cri-o://140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513" gracePeriod=45 Nov 02 18:13:44.224872 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.224559 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:13:44.382112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.381449 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-model-cache\") pod \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " Nov 02 18:13:44.382112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.381532 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tls-certs\") pod \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " Nov 02 18:13:44.382112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.381594 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tmp-dir\") pod \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " Nov 02 18:13:44.382112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.381660 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-home\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-home\") pod \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " Nov 02 18:13:44.382112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.381736 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-dshm\") pod \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " Nov 02 18:13:44.382112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.381770 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-kube-api-access-g98gp\" (UniqueName: \"kubernetes.io/projected/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-kube-api-access-g98gp\") pod \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\" (UID: \"e821ae21-b5ee-4eac-9bc6-c6de903df2bf\") " Nov 02 18:13:44.383537 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.383508 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-model-cache" pod "e821ae21-b5ee-4eac-9bc6-c6de903df2bf" (UID: "e821ae21-b5ee-4eac-9bc6-c6de903df2bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:44.391099 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.386415 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-home" pod "e821ae21-b5ee-4eac-9bc6-c6de903df2bf" (UID: "e821ae21-b5ee-4eac-9bc6-c6de903df2bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:44.391099 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.386575 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tmp-dir" pod "e821ae21-b5ee-4eac-9bc6-c6de903df2bf" (UID: "e821ae21-b5ee-4eac-9bc6-c6de903df2bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:44.415594 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.415532 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-dshm" pod "e821ae21-b5ee-4eac-9bc6-c6de903df2bf" (UID: "e821ae21-b5ee-4eac-9bc6-c6de903df2bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:44.415751 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.415603 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tls-certs" pod "e821ae21-b5ee-4eac-9bc6-c6de903df2bf" (UID: "e821ae21-b5ee-4eac-9bc6-c6de903df2bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:44.416072 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.416026 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-kube-api-access-g98gp" pod "e821ae21-b5ee-4eac-9bc6-c6de903df2bf" (UID: "e821ae21-b5ee-4eac-9bc6-c6de903df2bf"). InnerVolumeSpecName "kube-api-access-g98gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:44.484044 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.483852 4310 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-model-cache\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:44.484044 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.483892 4310 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tls-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:44.484044 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.483908 4310 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-tmp-dir\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:44.484044 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.483924 4310 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-home\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:44.484044 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.483938 4310 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-dshm\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:44.484044 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:44.483953 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g98gp\" (UniqueName: \"kubernetes.io/projected/e821ae21-b5ee-4eac-9bc6-c6de903df2bf-kube-api-access-g98gp\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:13:45.121992 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:45.121880 4310 generic.go:356] "Generic (PLEG): container finished" podID="e821ae21-b5ee-4eac-9bc6-c6de903df2bf" containerID="140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513" exitCode=0 Nov 02 18:13:45.122505 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:45.122024 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" event={"ID":"e821ae21-b5ee-4eac-9bc6-c6de903df2bf","Type":"ContainerDied","Data":"140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513"} Nov 02 18:13:45.122505 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:45.122089 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" event={"ID":"e821ae21-b5ee-4eac-9bc6-c6de903df2bf","Type":"ContainerDied","Data":"70be2bf70bd314acd64868836547c0fe15c4f12088911710135312a3129235ff"} Nov 02 18:13:45.122505 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:45.122119 4310 scope.go:122] "RemoveContainer" containerID="140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513" Nov 02 18:13:45.122505 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:45.122210 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv" Nov 02 18:13:45.138327 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:45.138302 4310 scope.go:122] "RemoveContainer" containerID="140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513" Nov 02 18:13:45.138685 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:13:45.138651 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513\": container with ID starting with 140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513 not found: ID does not exist" containerID="140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513" Nov 02 18:13:45.138770 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:45.138698 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513"} err="failed to get container status \"140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513\": rpc error: code = NotFound desc = could not find container \"140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513\": container with ID starting with 140401ecc3431361e387291f61d2c581203b6475fd7a32f2d2e7a6ac74db8513 not found: ID does not exist" Nov 02 18:13:45.162786 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:45.162743 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv"] Nov 02 18:13:45.188188 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:45.188133 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a6fb1955-kserve-8546cb74cc-nfrzv"] Nov 02 18:13:46.673746 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:46.673710 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e821ae21-b5ee-4eac-9bc6-c6de903df2bf" path="/var/lib/kubelet/pods/e821ae21-b5ee-4eac-9bc6-c6de903df2bf/volumes" Nov 02 18:13:48.916667 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:48.916633 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:13:49.024886 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:49.024850 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c"] Nov 02 18:13:49.027979 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:49.027946 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k7b9c"] Nov 02 18:13:50.681347 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:13:50.681307 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3850aa45-c7d3-4367-883b-177428028837" path="/var/lib/kubelet/pods/3850aa45-c7d3-4367-883b-177428028837/volumes" Nov 02 18:14:03.523098 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:14:03.523048 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:14:06.738937 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:06.738905 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:06.840254 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:06.840221 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:12.741239 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:12.741205 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:13.155720 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.155620 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt"] Nov 02 18:14:13.156044 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.156032 4310 cpu_manager.go:414] "containerMap: removing container" podUID="e821ae21-b5ee-4eac-9bc6-c6de903df2bf" containerName="main" Nov 02 18:14:13.156112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.156045 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e821ae21-b5ee-4eac-9bc6-c6de903df2bf" containerName="main" Nov 02 18:14:13.156155 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.156134 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="e821ae21-b5ee-4eac-9bc6-c6de903df2bf" containerName="main" Nov 02 18:14:13.160666 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.160645 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.164490 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.164467 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-082e58b0-openshift-default-dockercfg-6r574\"" Nov 02 18:14:13.171471 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.171436 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt"] Nov 02 18:14:13.213529 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.213493 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-socket\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.213705 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.213553 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-token\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.213705 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.213638 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istiod-ca-cert\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.213705 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.213674 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-podinfo\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.213865 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.213707 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-credential-socket\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.213865 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.213795 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-data\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.213865 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.213817 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-envoy\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.213865 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.213840 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-certs\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.213865 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.213861 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hpsd\" (UniqueName: \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-kube-api-access-9hpsd\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.223180 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.223154 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:13.317546 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.317507 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-podinfo\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.317867 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.317846 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-credential-socket\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.318433 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.318412 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-data\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.319231 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.319211 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-envoy\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.319817 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.319795 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-certs\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.320312 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.320293 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hpsd\" (UniqueName: \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-kube-api-access-9hpsd\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.320901 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.320880 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-socket\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.321091 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.319068 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-data\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.321730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.318324 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-credential-socket\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.321730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.320232 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-certs\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.321730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.321222 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-token\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.321730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.321645 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istiod-ca-cert\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.322436 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.322408 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istiod-ca-cert\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.322630 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.321360 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-socket\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.323850 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.323821 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-podinfo\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.324290 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.324269 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-envoy\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.325955 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.325930 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:13.329827 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.329776 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hpsd\" (UniqueName: \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-kube-api-access-9hpsd\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.330225 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.330200 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-token\") pod \"e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.478013 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.477977 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:13.801981 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.801781 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt"] Nov 02 18:14:13.802440 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.802403 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:14:13.802511 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.802482 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:14:13.802581 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:13.802531 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:14:14.302703 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:14.302666 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" event={"ID":"45c89976-f0be-404a-b1b5-bcdbbfe8aa28","Type":"ContainerStarted","Data":"6e4144330b904ae20057da23e35fbc4164c6c32ed83b97f8d8e23c563ee80b9d"} Nov 02 18:14:14.302939 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:14.302717 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" event={"ID":"45c89976-f0be-404a-b1b5-bcdbbfe8aa28","Type":"ContainerStarted","Data":"ca1b4820149e89740a0673e8a6c96f98beedecb31145e758c403680a640d3c54"} Nov 02 18:14:14.324608 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:14.324546 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" podStartSLOduration=1.3245327630000001 podStartE2EDuration="1.324532763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:14.324297233 +0000 UTC m=+2051.767379871" watchObservedRunningTime="2026-11-02 18:14:14.324532763 +0000 UTC m=+2051.767615399" Nov 02 18:14:14.478305 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:14.478266 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:14.483216 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:14.483189 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:15.307535 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:15.307507 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:15.308460 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:15.308437 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:15.862390 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:15.862327 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:15.962438 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:15.962402 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:17.104826 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.104790 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:17.226885 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.226852 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:17.361746 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.361659 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:17.911846 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.911817 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:17.947871 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.947812 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd"] Nov 02 18:14:17.951787 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.951768 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5"] Nov 02 18:14:17.951951 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.951900 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:17.955107 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.955068 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:17.962807 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.962783 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-082e58b0\"" Nov 02 18:14:17.964584 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.964563 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd"] Nov 02 18:14:17.965063 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.965038 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-082e58b0-dockercfg-lpbk5\"" Nov 02 18:14:17.968402 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:17.968378 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5"] Nov 02 18:14:18.026232 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.026201 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:18.073675 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.073639 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23763e7-ab70-4df3-97c9-9ddbe7540a80-plugins-config-volume\") pod \"payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5\" (UID: \"c23763e7-ab70-4df3-97c9-9ddbe7540a80\") " pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:18.073907 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.073703 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbph7\" (UniqueName: \"kubernetes.io/projected/c23763e7-ab70-4df3-97c9-9ddbe7540a80-kube-api-access-zbph7\") pod \"payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5\" (UID: \"c23763e7-ab70-4df3-97c9-9ddbe7540a80\") " pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:18.073907 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.073817 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccdffb7-6be0-4d28-813d-79d1eed589cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd\" (UID: \"1ccdffb7-6be0-4d28-813d-79d1eed589cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:18.074039 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.074000 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjh7\" (UniqueName: \"kubernetes.io/projected/1ccdffb7-6be0-4d28-813d-79d1eed589cd-kube-api-access-tsjh7\") pod \"payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd\" (UID: \"1ccdffb7-6be0-4d28-813d-79d1eed589cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:18.174970 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.174890 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjh7\" (UniqueName: \"kubernetes.io/projected/1ccdffb7-6be0-4d28-813d-79d1eed589cd-kube-api-access-tsjh7\") pod \"payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd\" (UID: \"1ccdffb7-6be0-4d28-813d-79d1eed589cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:18.174970 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.174948 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23763e7-ab70-4df3-97c9-9ddbe7540a80-plugins-config-volume\") pod \"payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5\" (UID: \"c23763e7-ab70-4df3-97c9-9ddbe7540a80\") " pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:18.174970 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.174976 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbph7\" (UniqueName: \"kubernetes.io/projected/c23763e7-ab70-4df3-97c9-9ddbe7540a80-kube-api-access-zbph7\") pod \"payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5\" (UID: \"c23763e7-ab70-4df3-97c9-9ddbe7540a80\") " pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:18.175559 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.175004 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccdffb7-6be0-4d28-813d-79d1eed589cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd\" (UID: \"1ccdffb7-6be0-4d28-813d-79d1eed589cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:18.175728 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.175708 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccdffb7-6be0-4d28-813d-79d1eed589cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd\" (UID: \"1ccdffb7-6be0-4d28-813d-79d1eed589cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:18.175765 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.175716 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23763e7-ab70-4df3-97c9-9ddbe7540a80-plugins-config-volume\") pod \"payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5\" (UID: \"c23763e7-ab70-4df3-97c9-9ddbe7540a80\") " pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:18.183091 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.183053 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbph7\" (UniqueName: \"kubernetes.io/projected/c23763e7-ab70-4df3-97c9-9ddbe7540a80-kube-api-access-zbph7\") pod \"payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5\" (UID: \"c23763e7-ab70-4df3-97c9-9ddbe7540a80\") " pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:18.183230 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.183057 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjh7\" (UniqueName: \"kubernetes.io/projected/1ccdffb7-6be0-4d28-813d-79d1eed589cd-kube-api-access-tsjh7\") pod \"payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd\" (UID: \"1ccdffb7-6be0-4d28-813d-79d1eed589cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:18.270452 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.270414 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:18.281764 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.281738 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:18.548370 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.548345 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd"] Nov 02 18:14:18.550378 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:14:18.550345 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ccdffb7_6be0_4d28_813d_79d1eed589cd.slice/crio-c8f7c150e01cfdfb220bf0279925983fa593272b82e0e8db10ebe354e548fb9f WatchSource:0}: Error finding container c8f7c150e01cfdfb220bf0279925983fa593272b82e0e8db10ebe354e548fb9f: Status 404 returned error can't find the container with id c8f7c150e01cfdfb220bf0279925983fa593272b82e0e8db10ebe354e548fb9f Nov 02 18:14:18.566017 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.565996 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5"] Nov 02 18:14:18.568175 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:14:18.568149 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23763e7_ab70_4df3_97c9_9ddbe7540a80.slice/crio-ec8f769cf80387229fa97aab76c9db351ddaadd56a7955ec8e7c5f9e25e5d29f WatchSource:0}: Error finding container ec8f769cf80387229fa97aab76c9db351ddaadd56a7955ec8e7c5f9e25e5d29f: Status 404 returned error can't find the container with id ec8f769cf80387229fa97aab76c9db351ddaadd56a7955ec8e7c5f9e25e5d29f Nov 02 18:14:18.777133 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:18.777092 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:19.329922 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:19.329624 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" event={"ID":"1ccdffb7-6be0-4d28-813d-79d1eed589cd","Type":"ContainerStarted","Data":"4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a"} Nov 02 18:14:19.329922 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:19.329728 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" event={"ID":"1ccdffb7-6be0-4d28-813d-79d1eed589cd","Type":"ContainerStarted","Data":"c8f7c150e01cfdfb220bf0279925983fa593272b82e0e8db10ebe354e548fb9f"} Nov 02 18:14:19.329922 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:19.329759 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:19.331685 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:19.331660 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" event={"ID":"c23763e7-ab70-4df3-97c9-9ddbe7540a80","Type":"ContainerStarted","Data":"76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31"} Nov 02 18:14:19.331797 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:19.331708 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:19.331797 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:19.331729 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" event={"ID":"c23763e7-ab70-4df3-97c9-9ddbe7540a80","Type":"ContainerStarted","Data":"ec8f769cf80387229fa97aab76c9db351ddaadd56a7955ec8e7c5f9e25e5d29f"} Nov 02 18:14:19.346359 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:19.346311 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" podStartSLOduration=2.346298689 podStartE2EDuration="2.346298689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:19.344857491 +0000 UTC m=+2056.787940127" watchObservedRunningTime="2026-11-02 18:14:19.346298689 +0000 UTC m=+2056.789381324" Nov 02 18:14:19.360551 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:19.360509 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" podStartSLOduration=2.360497165 podStartE2EDuration="2.360497165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:19.360118258 +0000 UTC m=+2056.803200896" watchObservedRunningTime="2026-11-02 18:14:19.360497165 +0000 UTC m=+2056.803579801" Nov 02 18:14:19.382003 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:19.381971 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:20.232832 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:20.232798 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:20.991495 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:20.991455 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:21.689356 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:21.689320 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:22.383883 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:22.383768 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:23.080328 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:23.080293 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:23.868510 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:23.868479 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:24.472791 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:24.472750 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:25.185300 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:25.185265 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:25.968632 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:25.968589 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:26.717983 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:26.717953 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:27.465938 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:27.465906 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:28.150236 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:28.150201 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:28.958939 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:28.958904 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:29.559813 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:29.559786 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:30.269626 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.269591 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:30.323796 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.322399 4310 patch_prober.go:29] interesting pod/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 18:14:30.323796 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.322489 4310 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 18:14:30.340340 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.340312 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:30.340630 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.340611 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:30.410485 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.410448 4310 generic.go:356] "Generic (PLEG): container finished" podID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerID="6e4144330b904ae20057da23e35fbc4164c6c32ed83b97f8d8e23c563ee80b9d" exitCode=0 Nov 02 18:14:30.410670 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.410515 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" event={"ID":"45c89976-f0be-404a-b1b5-bcdbbfe8aa28","Type":"ContainerDied","Data":"6e4144330b904ae20057da23e35fbc4164c6c32ed83b97f8d8e23c563ee80b9d"} Nov 02 18:14:30.410971 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.410948 4310 scope.go:122] "RemoveContainer" containerID="6e4144330b904ae20057da23e35fbc4164c6c32ed83b97f8d8e23c563ee80b9d" Nov 02 18:14:30.412267 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.412238 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:14:30.412387 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.412324 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:14:30.412387 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.412372 4310 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 18:14:30.971574 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:30.971537 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:31.417505 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:31.417422 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" event={"ID":"45c89976-f0be-404a-b1b5-bcdbbfe8aa28","Type":"ContainerStarted","Data":"dea4cb618f03c5a97cd31e6235f46ce3d6dd06314e1e898b0b9db6e7d93bb775"} Nov 02 18:14:31.478422 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:31.478381 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:31.478685 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:31.478664 4310 patch_prober.go:29] interesting pod/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 18:14:31.478731 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:31.478711 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 18:14:31.678239 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:31.678130 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:32.373551 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:32.373514 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:32.479149 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:32.479034 4310 patch_prober.go:29] interesting pod/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 18:14:32.479149 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:32.479117 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 18:14:33.101571 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:33.101532 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:33.478650 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:33.478617 4310 patch_prober.go:29] interesting pod/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 18:14:33.479069 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:33.478669 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 18:14:33.524465 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:14:33.524432 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:14:33.889389 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:33.889289 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:34.478686 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:34.478649 4310 patch_prober.go:29] interesting pod/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 18:14:34.479137 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:34.478704 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 18:14:34.590273 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:34.590234 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:35.270449 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:35.270410 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:35.479258 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:35.479218 4310 patch_prober.go:29] interesting pod/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 02 18:14:35.479646 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:35.479283 4310 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 02 18:14:35.974844 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:35.974806 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:36.482571 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:36.482535 4310 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:36.483113 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:36.483072 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:36.483845 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:36.483827 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:14:36.654305 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:36.654273 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:37.282371 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:37.282335 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:37.984863 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:37.984831 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:38.670227 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:38.670184 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:39.359506 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:39.359456 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:39.980725 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:39.980684 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:40.568858 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:40.568820 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:41.175221 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:41.175190 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:41.843295 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:41.843255 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:42.021139 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.021097 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc"] Nov 02 18:14:42.024151 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.024131 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:42.031617 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.031596 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:14:42.031893 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.031874 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pqf2d\"" Nov 02 18:14:42.034431 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.034388 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc"] Nov 02 18:14:42.080385 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.080358 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:42.080548 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.080391 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:42.080591 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.080542 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl5kr\" (UniqueName: \"kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr\") pod \"maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:42.181787 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.181698 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:42.181787 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.181736 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:42.181988 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.181804 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5kr\" (UniqueName: \"kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr\") pod \"maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:42.182468 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.182447 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:42.184226 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.184208 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:42.187832 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:14:42.187796 4310 projected.go:196] Error preparing data for projected volume kube-api-access-cl5kr for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 18:14:42.187987 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:14:42.187894 4310 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr podName:fd5c47c1-8cf5-4d3d-b01b-a2454930e33b nodeName:}" failed. No retries permitted until 2026-11-02 18:14:42.687871247 +0000 UTC m=+2080.130953878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cl5kr" (UniqueName: "kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr") pod "maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" (UID: "fd5c47c1-8cf5-4d3d-b01b-a2454930e33b") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 18:14:42.688651 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.688614 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5kr\" (UniqueName: \"kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr\") pod \"maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:42.693112 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:14:42.693057 4310 projected.go:196] Error preparing data for projected volume kube-api-access-cl5kr for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 18:14:42.693265 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:14:42.693143 4310 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr podName:fd5c47c1-8cf5-4d3d-b01b-a2454930e33b nodeName:}" failed. No retries permitted until 2026-11-02 18:14:43.693128405 +0000 UTC m=+2081.136211023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cl5kr" (UniqueName: "kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr") pod "maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" (UID: "fd5c47c1-8cf5-4d3d-b01b-a2454930e33b") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 18:14:42.749004 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:42.748966 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:43.377446 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:43.377407 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:43.698939 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:43.698853 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5kr\" (UniqueName: \"kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr\") pod \"maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:43.707057 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:43.707017 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl5kr\" (UniqueName: \"kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr\") pod \"maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:43.840179 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:43.840133 4310 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" secret="" err="secret \"maas-api-cleanup-dockercfg-pqf2d\" not found" Nov 02 18:14:43.840395 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:43.840197 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:44.047453 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:44.047420 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:44.123676 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:44.123650 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc"] Nov 02 18:14:44.126035 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:14:44.126009 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5c47c1_8cf5_4d3d_b01b_a2454930e33b.slice/crio-2fb09626f52eca12b5d2c60a871a2fbcd153e56a4fdf32618c91e5f4c186e64a WatchSource:0}: Error finding container 2fb09626f52eca12b5d2c60a871a2fbcd153e56a4fdf32618c91e5f4c186e64a: Status 404 returned error can't find the container with id 2fb09626f52eca12b5d2c60a871a2fbcd153e56a4fdf32618c91e5f4c186e64a Nov 02 18:14:44.489130 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:44.489088 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" event={"ID":"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b","Type":"ContainerStarted","Data":"d29c273fee008e503b84c456429aa2cf44c962675addfdae1742420fdccbec92"} Nov 02 18:14:44.489549 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:44.489153 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" event={"ID":"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b","Type":"ContainerStarted","Data":"2fb09626f52eca12b5d2c60a871a2fbcd153e56a4fdf32618c91e5f4c186e64a"} Nov 02 18:14:44.489549 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:44.489402 4310 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" secret="" err="secret \"maas-api-cleanup-dockercfg-pqf2d\" not found" Nov 02 18:14:44.506266 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:44.506209 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" podStartSLOduration=2.50619236 podStartE2EDuration="2.50619236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:44.504037186 +0000 UTC m=+2081.947119821" watchObservedRunningTime="2026-11-02 18:14:44.50619236 +0000 UTC m=+2081.949274996" Nov 02 18:14:44.662670 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:44.662568 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:45.369887 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:45.369839 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:45.496036 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:45.496000 4310 generic.go:356] "Generic (PLEG): container finished" podID="fd5c47c1-8cf5-4d3d-b01b-a2454930e33b" containerID="d29c273fee008e503b84c456429aa2cf44c962675addfdae1742420fdccbec92" exitCode=0 Nov 02 18:14:45.496531 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:45.496102 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" event={"ID":"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b","Type":"ContainerDied","Data":"d29c273fee008e503b84c456429aa2cf44c962675addfdae1742420fdccbec92"} Nov 02 18:14:45.972583 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:45.972546 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:46.567871 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:46.567832 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:46.657662 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:46.657638 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:46.738260 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:46.738228 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-openshift-ca\") pod \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " Nov 02 18:14:46.738428 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:46.738350 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-xks-ca\") pod \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " Nov 02 18:14:46.738428 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:46.738424 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr\" (UniqueName: \"kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr\") pod \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\" (UID: \"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b\") " Nov 02 18:14:46.738659 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:46.738634 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-openshift-ca" pod "fd5c47c1-8cf5-4d3d-b01b-a2454930e33b" (UID: "fd5c47c1-8cf5-4d3d-b01b-a2454930e33b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:14:46.740485 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:46.740453 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-xks-ca" pod "fd5c47c1-8cf5-4d3d-b01b-a2454930e33b" (UID: "fd5c47c1-8cf5-4d3d-b01b-a2454930e33b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:14:46.740485 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:46.740469 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr" pod "fd5c47c1-8cf5-4d3d-b01b-a2454930e33b" (UID: "fd5c47c1-8cf5-4d3d-b01b-a2454930e33b"). InnerVolumeSpecName "kube-api-access-cl5kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:46.840210 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:46.840122 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl5kr\" (UniqueName: \"kubernetes.io/projected/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-kube-api-access-cl5kr\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:46.840210 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:46.840156 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:46.840210 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:46.840167 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:47.282649 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:47.282614 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:47.508715 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:47.508683 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" Nov 02 18:14:47.508916 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:47.508681 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc" event={"ID":"fd5c47c1-8cf5-4d3d-b01b-a2454930e33b","Type":"ContainerDied","Data":"2fb09626f52eca12b5d2c60a871a2fbcd153e56a4fdf32618c91e5f4c186e64a"} Nov 02 18:14:47.508916 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:47.508815 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fb09626f52eca12b5d2c60a871a2fbcd153e56a4fdf32618c91e5f4c186e64a" Nov 02 18:14:47.948429 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:47.948389 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:48.567693 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:48.567650 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:48.952712 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:48.952616 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:49.176313 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:49.176275 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:49.849440 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:49.849411 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:50.545823 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:50.545792 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:51.062711 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:51.062676 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:51.762845 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:51.762802 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:52.154512 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:52.154428 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5"] Nov 02 18:14:52.167513 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:52.167486 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd"] Nov 02 18:14:52.226121 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:52.226051 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:52.333960 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:52.333931 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:52.566159 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:52.566127 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:53.130259 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:53.130223 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:53.277688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:53.277651 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:53.315455 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:53.315425 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc"] Nov 02 18:14:53.318688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:53.318649 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-082e58b0-z2dlc"] Nov 02 18:14:53.364151 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:53.364112 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:54.672996 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:54.672962 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5c47c1-8cf5-4d3d-b01b-a2454930e33b" path="/var/lib/kubelet/pods/fd5c47c1-8cf5-4d3d-b01b-a2454930e33b/volumes" Nov 02 18:14:57.155545 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.155504 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" podUID="c23763e7-ab70-4df3-97c9-9ddbe7540a80" containerName="payload-processing" containerID="cri-o://76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31" gracePeriod=25 Nov 02 18:14:57.168150 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.168120 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" podUID="1ccdffb7-6be0-4d28-813d-79d1eed589cd" containerName="payload-pre-processing" containerID="cri-o://4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a" gracePeriod=25 Nov 02 18:14:57.418494 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.418436 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:57.448177 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.448147 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:57.564199 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.564163 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23763e7-ab70-4df3-97c9-9ddbe7540a80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23763e7-ab70-4df3-97c9-9ddbe7540a80-plugins-config-volume\") pod \"c23763e7-ab70-4df3-97c9-9ddbe7540a80\" (UID: \"c23763e7-ab70-4df3-97c9-9ddbe7540a80\") " Nov 02 18:14:57.564352 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.564238 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ccdffb7-6be0-4d28-813d-79d1eed589cd-kube-api-access-tsjh7\" (UniqueName: \"kubernetes.io/projected/1ccdffb7-6be0-4d28-813d-79d1eed589cd-kube-api-access-tsjh7\") pod \"1ccdffb7-6be0-4d28-813d-79d1eed589cd\" (UID: \"1ccdffb7-6be0-4d28-813d-79d1eed589cd\") " Nov 02 18:14:57.564352 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.564267 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ccdffb7-6be0-4d28-813d-79d1eed589cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccdffb7-6be0-4d28-813d-79d1eed589cd-plugins-config-volume\") pod \"1ccdffb7-6be0-4d28-813d-79d1eed589cd\" (UID: \"1ccdffb7-6be0-4d28-813d-79d1eed589cd\") " Nov 02 18:14:57.564352 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.564319 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23763e7-ab70-4df3-97c9-9ddbe7540a80-kube-api-access-zbph7\" (UniqueName: \"kubernetes.io/projected/c23763e7-ab70-4df3-97c9-9ddbe7540a80-kube-api-access-zbph7\") pod \"c23763e7-ab70-4df3-97c9-9ddbe7540a80\" (UID: \"c23763e7-ab70-4df3-97c9-9ddbe7540a80\") " Nov 02 18:14:57.564670 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.564644 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23763e7-ab70-4df3-97c9-9ddbe7540a80-plugins-config-volume" pod "c23763e7-ab70-4df3-97c9-9ddbe7540a80" (UID: "c23763e7-ab70-4df3-97c9-9ddbe7540a80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:14:57.564712 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.564646 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ccdffb7-6be0-4d28-813d-79d1eed589cd-plugins-config-volume" pod "1ccdffb7-6be0-4d28-813d-79d1eed589cd" (UID: "1ccdffb7-6be0-4d28-813d-79d1eed589cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:14:57.566478 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.566458 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23763e7-ab70-4df3-97c9-9ddbe7540a80-kube-api-access-zbph7" pod "c23763e7-ab70-4df3-97c9-9ddbe7540a80" (UID: "c23763e7-ab70-4df3-97c9-9ddbe7540a80"). InnerVolumeSpecName "kube-api-access-zbph7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:57.566534 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.566484 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccdffb7-6be0-4d28-813d-79d1eed589cd-kube-api-access-tsjh7" pod "1ccdffb7-6be0-4d28-813d-79d1eed589cd" (UID: "1ccdffb7-6be0-4d28-813d-79d1eed589cd"). InnerVolumeSpecName "kube-api-access-tsjh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:57.574926 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.574899 4310 generic.go:356] "Generic (PLEG): container finished" podID="1ccdffb7-6be0-4d28-813d-79d1eed589cd" containerID="4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a" exitCode=0 Nov 02 18:14:57.575090 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.574941 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" Nov 02 18:14:57.575090 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.574980 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" event={"ID":"1ccdffb7-6be0-4d28-813d-79d1eed589cd","Type":"ContainerDied","Data":"4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a"} Nov 02 18:14:57.575090 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.575035 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd" event={"ID":"1ccdffb7-6be0-4d28-813d-79d1eed589cd","Type":"ContainerDied","Data":"c8f7c150e01cfdfb220bf0279925983fa593272b82e0e8db10ebe354e548fb9f"} Nov 02 18:14:57.575090 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.575073 4310 scope.go:122] "RemoveContainer" containerID="4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a" Nov 02 18:14:57.576378 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.576360 4310 generic.go:356] "Generic (PLEG): container finished" podID="c23763e7-ab70-4df3-97c9-9ddbe7540a80" containerID="76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31" exitCode=0 Nov 02 18:14:57.576465 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.576407 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" Nov 02 18:14:57.576465 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.576449 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" event={"ID":"c23763e7-ab70-4df3-97c9-9ddbe7540a80","Type":"ContainerDied","Data":"76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31"} Nov 02 18:14:57.576573 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.576500 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5" event={"ID":"c23763e7-ab70-4df3-97c9-9ddbe7540a80","Type":"ContainerDied","Data":"ec8f769cf80387229fa97aab76c9db351ddaadd56a7955ec8e7c5f9e25e5d29f"} Nov 02 18:14:57.585961 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.585943 4310 scope.go:122] "RemoveContainer" containerID="4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a" Nov 02 18:14:57.586308 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:14:57.586217 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a\": container with ID starting with 4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a not found: ID does not exist" containerID="4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a" Nov 02 18:14:57.586308 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.586241 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a"} err="failed to get container status \"4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a\": rpc error: code = NotFound desc = could not find container \"4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a\": container with ID starting with 4ecbc49264381a757a03267c917c15bea9a36320c58a0e495c107ba9a8e6bf3a not found: ID does not exist" Nov 02 18:14:57.586308 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.586257 4310 scope.go:122] "RemoveContainer" containerID="76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31" Nov 02 18:14:57.593893 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.593874 4310 scope.go:122] "RemoveContainer" containerID="76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31" Nov 02 18:14:57.594160 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:14:57.594139 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31\": container with ID starting with 76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31 not found: ID does not exist" containerID="76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31" Nov 02 18:14:57.594239 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.594165 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31"} err="failed to get container status \"76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31\": rpc error: code = NotFound desc = could not find container \"76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31\": container with ID starting with 76d6d754d28cee0b7b9f4052ee3b6366e4a30ff6dab6e6b40682757129643d31 not found: ID does not exist" Nov 02 18:14:57.598910 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.598890 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd"] Nov 02 18:14:57.600721 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.600699 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-082e58b0-dc474d599-8crrd"] Nov 02 18:14:57.610387 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.610369 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5"] Nov 02 18:14:57.614668 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.614641 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-082e58b0-57c5869bc-wkkd5"] Nov 02 18:14:57.665724 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.665693 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsjh7\" (UniqueName: \"kubernetes.io/projected/1ccdffb7-6be0-4d28-813d-79d1eed589cd-kube-api-access-tsjh7\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:57.665724 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.665721 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ccdffb7-6be0-4d28-813d-79d1eed589cd-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:57.665962 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.665741 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbph7\" (UniqueName: \"kubernetes.io/projected/c23763e7-ab70-4df3-97c9-9ddbe7540a80-kube-api-access-zbph7\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:57.665962 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:57.665752 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23763e7-ab70-4df3-97c9-9ddbe7540a80-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:14:58.461505 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:58.461407 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt"] Nov 02 18:14:58.462004 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:58.461692 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" containerID="cri-o://dea4cb618f03c5a97cd31e6235f46ce3d6dd06314e1e898b0b9db6e7d93bb775" gracePeriod=30 Nov 02 18:14:58.488300 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:58.488270 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:14:58.672550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:58.672514 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ccdffb7-6be0-4d28-813d-79d1eed589cd" path="/var/lib/kubelet/pods/1ccdffb7-6be0-4d28-813d-79d1eed589cd/volumes" Nov 02 18:14:58.673065 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:58.673048 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23763e7-ab70-4df3-97c9-9ddbe7540a80" path="/var/lib/kubelet/pods/c23763e7-ab70-4df3-97c9-9ddbe7540a80/volumes" Nov 02 18:14:59.767839 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:14:59.767801 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:00.136481 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.136386 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq"] Nov 02 18:15:00.136968 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.136935 4310 cpu_manager.go:414] "containerMap: removing container" podUID="1ccdffb7-6be0-4d28-813d-79d1eed589cd" containerName="payload-pre-processing" Nov 02 18:15:00.136968 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.136957 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ccdffb7-6be0-4d28-813d-79d1eed589cd" containerName="payload-pre-processing" Nov 02 18:15:00.137232 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.136976 4310 cpu_manager.go:414] "containerMap: removing container" podUID="c23763e7-ab70-4df3-97c9-9ddbe7540a80" containerName="payload-processing" Nov 02 18:15:00.137232 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.136984 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23763e7-ab70-4df3-97c9-9ddbe7540a80" containerName="payload-processing" Nov 02 18:15:00.137232 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.137004 4310 cpu_manager.go:414] "containerMap: removing container" podUID="fd5c47c1-8cf5-4d3d-b01b-a2454930e33b" containerName="revoke-keys" Nov 02 18:15:00.137232 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.137011 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5c47c1-8cf5-4d3d-b01b-a2454930e33b" containerName="revoke-keys" Nov 02 18:15:00.137232 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.137121 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5c47c1-8cf5-4d3d-b01b-a2454930e33b" containerName="revoke-keys" Nov 02 18:15:00.137232 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.137142 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ccdffb7-6be0-4d28-813d-79d1eed589cd" containerName="payload-pre-processing" Nov 02 18:15:00.137232 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.137151 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23763e7-ab70-4df3-97c9-9ddbe7540a80" containerName="payload-processing" Nov 02 18:15:00.140557 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.140533 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" Nov 02 18:15:00.151934 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.151908 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqh67\"" Nov 02 18:15:00.153486 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.153440 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq"] Nov 02 18:15:00.195383 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.195334 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:00.291220 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.291177 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcxb2\" (UniqueName: \"kubernetes.io/projected/a667dd99-0167-4b7e-a43e-dd2583bba691-kube-api-access-zcxb2\") pod \"maas-api-key-cleanup-29894055-x5pjq\" (UID: \"a667dd99-0167-4b7e-a43e-dd2583bba691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" Nov 02 18:15:00.291505 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.291485 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a667dd99-0167-4b7e-a43e-dd2583bba691-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-x5pjq\" (UID: \"a667dd99-0167-4b7e-a43e-dd2583bba691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" Nov 02 18:15:00.299471 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.299433 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:00.395994 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.395902 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxb2\" (UniqueName: \"kubernetes.io/projected/a667dd99-0167-4b7e-a43e-dd2583bba691-kube-api-access-zcxb2\") pod \"maas-api-key-cleanup-29894055-x5pjq\" (UID: \"a667dd99-0167-4b7e-a43e-dd2583bba691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" Nov 02 18:15:00.395994 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.395949 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a667dd99-0167-4b7e-a43e-dd2583bba691-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-x5pjq\" (UID: \"a667dd99-0167-4b7e-a43e-dd2583bba691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" Nov 02 18:15:00.396586 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.396567 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a667dd99-0167-4b7e-a43e-dd2583bba691-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-x5pjq\" (UID: \"a667dd99-0167-4b7e-a43e-dd2583bba691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" Nov 02 18:15:00.406565 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.406538 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxb2\" (UniqueName: \"kubernetes.io/projected/a667dd99-0167-4b7e-a43e-dd2583bba691-kube-api-access-zcxb2\") pod \"maas-api-key-cleanup-29894055-x5pjq\" (UID: \"a667dd99-0167-4b7e-a43e-dd2583bba691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" Nov 02 18:15:00.456445 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.456406 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" Nov 02 18:15:00.735551 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:00.735522 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq"] Nov 02 18:15:00.737983 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:15:00.737943 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda667dd99_0167_4b7e_a43e_dd2583bba691.slice/crio-44a72af83aa5c1912ea41f89ae3d27c5b4aa7c28e47a04ef37730d820ac13bf0 WatchSource:0}: Error finding container 44a72af83aa5c1912ea41f89ae3d27c5b4aa7c28e47a04ef37730d820ac13bf0: Status 404 returned error can't find the container with id 44a72af83aa5c1912ea41f89ae3d27c5b4aa7c28e47a04ef37730d820ac13bf0 Nov 02 18:15:01.605653 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:01.605615 4310 generic.go:356] "Generic (PLEG): container finished" podID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerID="ec9b079cd6bd53c6000b617a5e11f08fb48c8aae5003441ae6cdfa1a1113670c" exitCode=60 Nov 02 18:15:01.606150 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:01.605704 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" event={"ID":"a667dd99-0167-4b7e-a43e-dd2583bba691","Type":"ContainerDied","Data":"ec9b079cd6bd53c6000b617a5e11f08fb48c8aae5003441ae6cdfa1a1113670c"} Nov 02 18:15:01.606150 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:01.605749 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" event={"ID":"a667dd99-0167-4b7e-a43e-dd2583bba691","Type":"ContainerStarted","Data":"44a72af83aa5c1912ea41f89ae3d27c5b4aa7c28e47a04ef37730d820ac13bf0"} Nov 02 18:15:01.606150 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:01.605908 4310 scope.go:122] "RemoveContainer" containerID="ec9b079cd6bd53c6000b617a5e11f08fb48c8aae5003441ae6cdfa1a1113670c" Nov 02 18:15:02.614632 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:02.614548 4310 generic.go:356] "Generic (PLEG): container finished" podID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerID="afe708b3f5e18f6c026da7748a908cd2b78e93ea5bd373a9830d25608cbc7b46" exitCode=60 Nov 02 18:15:02.615043 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:02.614623 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" event={"ID":"a667dd99-0167-4b7e-a43e-dd2583bba691","Type":"ContainerDied","Data":"afe708b3f5e18f6c026da7748a908cd2b78e93ea5bd373a9830d25608cbc7b46"} Nov 02 18:15:02.615043 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:02.614682 4310 scope.go:122] "RemoveContainer" containerID="ec9b079cd6bd53c6000b617a5e11f08fb48c8aae5003441ae6cdfa1a1113670c" Nov 02 18:15:02.615043 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:02.614987 4310 scope.go:122] "RemoveContainer" containerID="afe708b3f5e18f6c026da7748a908cd2b78e93ea5bd373a9830d25608cbc7b46" Nov 02 18:15:02.615301 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:15:02.615277 4310 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-x5pjq_odh-ai-gateway-infra(a667dd99-0167-4b7e-a43e-dd2583bba691)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" Nov 02 18:15:03.523484 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:15:03.523439 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:15:03.613989 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.613938 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:03.628880 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.628842 4310 generic.go:356] "Generic (PLEG): container finished" podID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerID="dea4cb618f03c5a97cd31e6235f46ce3d6dd06314e1e898b0b9db6e7d93bb775" exitCode=0 Nov 02 18:15:03.629412 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.628965 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" event={"ID":"45c89976-f0be-404a-b1b5-bcdbbfe8aa28","Type":"ContainerDied","Data":"dea4cb618f03c5a97cd31e6235f46ce3d6dd06314e1e898b0b9db6e7d93bb775"} Nov 02 18:15:03.629412 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.629092 4310 scope.go:122] "RemoveContainer" containerID="6e4144330b904ae20057da23e35fbc4164c6c32ed83b97f8d8e23c563ee80b9d" Nov 02 18:15:03.632503 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.632483 4310 scope.go:122] "RemoveContainer" containerID="afe708b3f5e18f6c026da7748a908cd2b78e93ea5bd373a9830d25608cbc7b46" Nov 02 18:15:03.632717 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:15:03.632697 4310 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-x5pjq_odh-ai-gateway-infra(a667dd99-0167-4b7e-a43e-dd2583bba691)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" Nov 02 18:15:03.706679 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.706649 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:03.742955 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.742931 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:15:03.832521 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.832488 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-data\") pod \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " Nov 02 18:15:03.832521 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.832525 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-token\" (UniqueName: \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-token\") pod \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " Nov 02 18:15:03.832756 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.832546 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-credential-socket\") pod \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " Nov 02 18:15:03.832756 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.832580 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-envoy\") pod \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " Nov 02 18:15:03.832756 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.832606 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-kube-api-access-9hpsd\" (UniqueName: \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-kube-api-access-9hpsd\") pod \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " Nov 02 18:15:03.832756 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.832690 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-certs\") pod \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " Nov 02 18:15:03.832756 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.832732 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-podinfo\") pod \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " Nov 02 18:15:03.833068 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.832789 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istiod-ca-cert\") pod \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " Nov 02 18:15:03.833068 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.832810 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-data" pod "45c89976-f0be-404a-b1b5-bcdbbfe8aa28" (UID: "45c89976-f0be-404a-b1b5-bcdbbfe8aa28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:03.833068 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.832825 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-socket\") pod \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\" (UID: \"45c89976-f0be-404a-b1b5-bcdbbfe8aa28\") " Nov 02 18:15:03.833068 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.832895 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-credential-socket" pod "45c89976-f0be-404a-b1b5-bcdbbfe8aa28" (UID: "45c89976-f0be-404a-b1b5-bcdbbfe8aa28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:03.833068 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.833031 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-certs" pod "45c89976-f0be-404a-b1b5-bcdbbfe8aa28" (UID: "45c89976-f0be-404a-b1b5-bcdbbfe8aa28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:03.833068 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.833060 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-socket" pod "45c89976-f0be-404a-b1b5-bcdbbfe8aa28" (UID: "45c89976-f0be-404a-b1b5-bcdbbfe8aa28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:03.833405 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.833310 4310 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:03.833405 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.833343 4310 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-data\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:03.833405 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.833358 4310 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-credential-socket\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:03.833405 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.833352 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istiod-ca-cert" pod "45c89976-f0be-404a-b1b5-bcdbbfe8aa28" (UID: "45c89976-f0be-404a-b1b5-bcdbbfe8aa28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:03.833405 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.833374 4310 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-workload-certs\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:03.834925 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.834903 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-envoy" pod "45c89976-f0be-404a-b1b5-bcdbbfe8aa28" (UID: "45c89976-f0be-404a-b1b5-bcdbbfe8aa28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:03.835136 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.835117 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-podinfo" pod "45c89976-f0be-404a-b1b5-bcdbbfe8aa28" (UID: "45c89976-f0be-404a-b1b5-bcdbbfe8aa28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:15:03.835219 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.835180 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-token" pod "45c89976-f0be-404a-b1b5-bcdbbfe8aa28" (UID: "45c89976-f0be-404a-b1b5-bcdbbfe8aa28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:03.835261 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.835249 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-kube-api-access-9hpsd" pod "45c89976-f0be-404a-b1b5-bcdbbfe8aa28" (UID: "45c89976-f0be-404a-b1b5-bcdbbfe8aa28"). InnerVolumeSpecName "kube-api-access-9hpsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:03.934073 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.934034 4310 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-token\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:03.934073 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.934071 4310 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-envoy\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:03.934321 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.934119 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hpsd\" (UniqueName: \"kubernetes.io/projected/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-kube-api-access-9hpsd\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:03.934321 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.934134 4310 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istio-podinfo\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:03.934321 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:03.934148 4310 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c89976-f0be-404a-b1b5-bcdbbfe8aa28-istiod-ca-cert\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:04.638094 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:04.638055 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" Nov 02 18:15:04.638549 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:04.638052 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt" event={"ID":"45c89976-f0be-404a-b1b5-bcdbbfe8aa28","Type":"ContainerDied","Data":"ca1b4820149e89740a0673e8a6c96f98beedecb31145e758c403680a640d3c54"} Nov 02 18:15:04.638549 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:04.638193 4310 scope.go:122] "RemoveContainer" containerID="dea4cb618f03c5a97cd31e6235f46ce3d6dd06314e1e898b0b9db6e7d93bb775" Nov 02 18:15:04.662942 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:04.662908 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt"] Nov 02 18:15:04.673997 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:04.673969 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-082e58b0-openshift-default-5d765f5b8f-zbcxt"] Nov 02 18:15:04.785127 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:04.785063 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:04.886660 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:04.886623 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:05.107608 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:05.107572 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:05.560904 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:05.560862 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:05.597323 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:05.597293 4310 scope.go:122] "RemoveContainer" containerID="25cc1beec9c512df134210fe641ecf9e5ca2a36d6f4f9994bd1ac71459382466" Nov 02 18:15:05.608596 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:05.608558 4310 scope.go:122] "RemoveContainer" containerID="a9619da3aaf158c000e8f392ab3f9a334bc1310a6f6d9034d9567c58d3b92228" Nov 02 18:15:05.621089 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:05.621053 4310 scope.go:122] "RemoveContainer" containerID="fc7af6e354ea1c0ed78bc1ce35623e2d9c03a2af1ef95386ae125b804ba4a313" Nov 02 18:15:06.250007 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.249974 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:06.672046 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.671963 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" path="/var/lib/kubelet/pods/45c89976-f0be-404a-b1b5-bcdbbfe8aa28/volumes" Nov 02 18:15:06.856634 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.856597 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:06.885368 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.885332 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6"] Nov 02 18:15:06.885735 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.885722 4310 cpu_manager.go:414] "containerMap: removing container" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" Nov 02 18:15:06.885796 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.885736 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" Nov 02 18:15:06.885861 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.885806 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" Nov 02 18:15:06.890591 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.890569 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:15:06.904345 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.904310 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz"] Nov 02 18:15:06.905226 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.905202 4310 cpu_manager.go:414] "containerMap: removing container" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" Nov 02 18:15:06.905391 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.905377 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" Nov 02 18:15:06.905682 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.905668 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c89976-f0be-404a-b1b5-bcdbbfe8aa28" containerName="istio-proxy" Nov 02 18:15:06.909905 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.909884 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:15:06.910656 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.910634 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-5d4d8d\"" Nov 02 18:15:06.915030 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.914999 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6"] Nov 02 18:15:06.917626 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.917598 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz"] Nov 02 18:15:06.918461 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.918258 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-5d4d8d-dockercfg-rjk5k\"" Nov 02 18:15:06.960151 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.960050 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znt69\" (UniqueName: \"kubernetes.io/projected/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-kube-api-access-znt69\") pod \"payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6\" (UID: \"84df1dca-0d2e-4488-af2a-2dbdb282b3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:15:06.960299 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.960174 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7z5t\" (UniqueName: \"kubernetes.io/projected/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-kube-api-access-b7z5t\") pod \"payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz\" (UID: \"f504cfbd-f914-4eb0-bd9c-fc48a881a87b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:15:06.960299 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.960196 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6\" (UID: \"84df1dca-0d2e-4488-af2a-2dbdb282b3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:15:06.960299 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.960244 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz\" (UID: \"f504cfbd-f914-4eb0-bd9c-fc48a881a87b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:15:06.967271 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:06.967247 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:07.061616 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.061579 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz\" (UID: \"f504cfbd-f914-4eb0-bd9c-fc48a881a87b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:15:07.061815 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.061676 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znt69\" (UniqueName: \"kubernetes.io/projected/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-kube-api-access-znt69\") pod \"payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6\" (UID: \"84df1dca-0d2e-4488-af2a-2dbdb282b3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:15:07.061815 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.061719 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7z5t\" (UniqueName: \"kubernetes.io/projected/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-kube-api-access-b7z5t\") pod \"payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz\" (UID: \"f504cfbd-f914-4eb0-bd9c-fc48a881a87b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:15:07.062050 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.061901 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6\" (UID: \"84df1dca-0d2e-4488-af2a-2dbdb282b3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:15:07.062508 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.062478 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz\" (UID: \"f504cfbd-f914-4eb0-bd9c-fc48a881a87b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:15:07.062684 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.062662 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6\" (UID: \"84df1dca-0d2e-4488-af2a-2dbdb282b3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:15:07.076067 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.076040 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7z5t\" (UniqueName: \"kubernetes.io/projected/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-kube-api-access-b7z5t\") pod \"payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz\" (UID: \"f504cfbd-f914-4eb0-bd9c-fc48a881a87b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:15:07.076524 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.076501 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znt69\" (UniqueName: \"kubernetes.io/projected/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-kube-api-access-znt69\") pod \"payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6\" (UID: \"84df1dca-0d2e-4488-af2a-2dbdb282b3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:15:07.218566 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.218532 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:15:07.228721 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.228695 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:15:07.522773 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.522301 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6"] Nov 02 18:15:07.573651 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.573629 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz"] Nov 02 18:15:07.575586 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:15:07.575561 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf504cfbd_f914_4eb0_bd9c_fc48a881a87b.slice/crio-467eb23d5f2d43178c0997190765071af307ef55c2cd4cc86177c099cb93cf94 WatchSource:0}: Error finding container 467eb23d5f2d43178c0997190765071af307ef55c2cd4cc86177c099cb93cf94: Status 404 returned error can't find the container with id 467eb23d5f2d43178c0997190765071af307ef55c2cd4cc86177c099cb93cf94 Nov 02 18:15:07.659550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.659512 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" event={"ID":"84df1dca-0d2e-4488-af2a-2dbdb282b3c6","Type":"ContainerStarted","Data":"be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec"} Nov 02 18:15:07.659750 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.659580 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" event={"ID":"84df1dca-0d2e-4488-af2a-2dbdb282b3c6","Type":"ContainerStarted","Data":"68d46c329bab903b76e5fbca27fc2fda128442ec17989d9b9f681f1beac59c38"} Nov 02 18:15:07.659750 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.659622 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:15:07.665007 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.664978 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" event={"ID":"f504cfbd-f914-4eb0-bd9c-fc48a881a87b","Type":"ContainerStarted","Data":"fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6"} Nov 02 18:15:07.665177 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.665029 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" event={"ID":"f504cfbd-f914-4eb0-bd9c-fc48a881a87b","Type":"ContainerStarted","Data":"467eb23d5f2d43178c0997190765071af307ef55c2cd4cc86177c099cb93cf94"} Nov 02 18:15:07.665177 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.665071 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:15:07.676816 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.676765 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" podStartSLOduration=1.676747883 podStartE2EDuration="1.676747883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:07.674986617 +0000 UTC m=+2105.118069255" watchObservedRunningTime="2026-11-02 18:15:07.676747883 +0000 UTC m=+2105.119830523" Nov 02 18:15:07.692382 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.692337 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" podStartSLOduration=1.692324758 podStartE2EDuration="1.692324758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:07.691616718 +0000 UTC m=+2105.134699358" watchObservedRunningTime="2026-11-02 18:15:07.692324758 +0000 UTC m=+2105.135407393" Nov 02 18:15:07.863614 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:07.863495 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:08.477873 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:08.477840 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:09.181434 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:09.181342 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:09.878417 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:09.878380 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:10.579565 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:10.579528 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:11.307682 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:11.307626 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:12.064488 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:12.064458 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:12.665424 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:12.665377 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:12.668941 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:12.668923 4310 scope.go:122] "RemoveContainer" containerID="afe708b3f5e18f6c026da7748a908cd2b78e93ea5bd373a9830d25608cbc7b46" Nov 02 18:15:13.340178 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:13.340145 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:13.699933 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:13.699841 4310 generic.go:356] "Generic (PLEG): container finished" podID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerID="dcb8693a2d3f532825c86f82e2c4ff5b70c2e09e90c456bedd177859676ade6a" exitCode=60 Nov 02 18:15:13.699933 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:13.699893 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" event={"ID":"a667dd99-0167-4b7e-a43e-dd2583bba691","Type":"ContainerDied","Data":"dcb8693a2d3f532825c86f82e2c4ff5b70c2e09e90c456bedd177859676ade6a"} Nov 02 18:15:13.700193 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:13.699955 4310 scope.go:122] "RemoveContainer" containerID="afe708b3f5e18f6c026da7748a908cd2b78e93ea5bd373a9830d25608cbc7b46" Nov 02 18:15:13.700367 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:13.700343 4310 scope.go:122] "RemoveContainer" containerID="dcb8693a2d3f532825c86f82e2c4ff5b70c2e09e90c456bedd177859676ade6a" Nov 02 18:15:13.700630 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:15:13.700605 4310 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-x5pjq_odh-ai-gateway-infra(a667dd99-0167-4b7e-a43e-dd2583bba691)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" Nov 02 18:15:14.195141 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:14.195103 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:14.726434 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:14.726395 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq"] Nov 02 18:15:14.868640 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:14.868614 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" Nov 02 18:15:14.948011 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:14.947976 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a667dd99-0167-4b7e-a43e-dd2583bba691-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a667dd99-0167-4b7e-a43e-dd2583bba691-maas-api-service-ca\") pod \"a667dd99-0167-4b7e-a43e-dd2583bba691\" (UID: \"a667dd99-0167-4b7e-a43e-dd2583bba691\") " Nov 02 18:15:14.948229 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:14.948070 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a667dd99-0167-4b7e-a43e-dd2583bba691-kube-api-access-zcxb2\" (UniqueName: \"kubernetes.io/projected/a667dd99-0167-4b7e-a43e-dd2583bba691-kube-api-access-zcxb2\") pod \"a667dd99-0167-4b7e-a43e-dd2583bba691\" (UID: \"a667dd99-0167-4b7e-a43e-dd2583bba691\") " Nov 02 18:15:14.948540 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:14.948508 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a667dd99-0167-4b7e-a43e-dd2583bba691-maas-api-service-ca" pod "a667dd99-0167-4b7e-a43e-dd2583bba691" (UID: "a667dd99-0167-4b7e-a43e-dd2583bba691"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:14.950259 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:14.950227 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a667dd99-0167-4b7e-a43e-dd2583bba691-kube-api-access-zcxb2" pod "a667dd99-0167-4b7e-a43e-dd2583bba691" (UID: "a667dd99-0167-4b7e-a43e-dd2583bba691"). InnerVolumeSpecName "kube-api-access-zcxb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:14.967521 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:14.967454 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:15.049110 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:15.048998 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcxb2\" (UniqueName: \"kubernetes.io/projected/a667dd99-0167-4b7e-a43e-dd2583bba691-kube-api-access-zcxb2\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:15.049110 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:15.049029 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a667dd99-0167-4b7e-a43e-dd2583bba691-maas-api-service-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:15:15.656296 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:15.656263 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:15.723479 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:15.723448 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" Nov 02 18:15:15.723479 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:15.723472 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq" event={"ID":"a667dd99-0167-4b7e-a43e-dd2583bba691","Type":"ContainerDied","Data":"44a72af83aa5c1912ea41f89ae3d27c5b4aa7c28e47a04ef37730d820ac13bf0"} Nov 02 18:15:15.723770 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:15.723563 4310 scope.go:122] "RemoveContainer" containerID="dcb8693a2d3f532825c86f82e2c4ff5b70c2e09e90c456bedd177859676ade6a" Nov 02 18:15:15.752153 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:15.752119 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq"] Nov 02 18:15:15.756172 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:15.756142 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-x5pjq"] Nov 02 18:15:16.280373 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:16.280299 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:16.672421 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:16.672315 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" path="/var/lib/kubelet/pods/a667dd99-0167-4b7e-a43e-dd2583bba691/volumes" Nov 02 18:15:16.975832 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:16.975795 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:17.696256 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:17.696225 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:18.408605 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:18.408565 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:18.557904 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:18.557874 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:18.674294 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:18.674212 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:15:18.674294 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:18.674292 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:15:19.226875 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:19.226838 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:19.854261 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:19.854226 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:20.654675 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:20.654643 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:21.258465 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:21.258430 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:21.944438 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:21.944401 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:22.646377 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:22.646342 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:23.175551 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:23.175518 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:23.876687 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:23.876650 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:24.470842 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:24.470806 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:25.081996 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:25.081953 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:25.849374 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:25.849338 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:26.480610 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:26.480579 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:27.168182 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:27.168139 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:27.952909 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:27.952878 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:28.561323 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:28.561282 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:29.157966 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:29.157931 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:29.771401 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:29.771364 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:30.362712 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:30.362654 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:30.972157 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:30.972117 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:31.589393 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:31.589350 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:32.279439 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:32.279407 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:33.065238 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:33.065194 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:33.525852 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:15:33.525812 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:15:33.678606 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:33.678570 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:34.352105 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:34.352048 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:34.959987 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:34.959953 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:35.573888 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:35.573851 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:36.258904 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:36.258869 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:36.878972 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:36.878937 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:37.473409 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:37.473372 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:38.066558 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:38.066524 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:38.675806 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:38.675771 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:39.283400 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:39.283365 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:39.869547 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:39.869515 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:40.679287 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:40.679256 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:41.453748 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:41.453711 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:42.152292 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:42.152254 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:42.747676 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:42.747646 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:43.362309 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:43.362276 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:43.961902 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:43.961869 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:44.561693 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:44.561638 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:45.165625 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:45.165577 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:45.879752 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:45.879717 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:46.666843 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:46.666810 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:47.356546 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:47.356512 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:48.005538 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:48.005502 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:48.657619 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:48.657584 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:48.998867 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:48.998821 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:49.098280 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:49.098241 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:49.323929 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:49.323835 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:49.960293 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:49.960258 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:50.568462 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:50.568414 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:51.277910 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:51.277875 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:52.062958 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:52.062926 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:52.667159 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:52.667112 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:53.276753 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:53.276719 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:53.949965 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:53.949932 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:54.574590 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:54.574556 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:55.276176 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:55.276125 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:55.964513 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:55.964478 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:56.298145 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:56.298017 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:56.677886 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:56.677809 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:57.388053 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:57.388021 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:58.036676 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:58.036637 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:58.680111 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:58.680060 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:59.328815 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:59.328786 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:15:59.974067 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:15:59.974031 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:00.674829 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:00.674796 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:01.358749 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:01.358718 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:02.068385 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:02.068355 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:02.880167 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:02.880129 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:03.508253 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:03.508203 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:03.523952 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:16:03.523912 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:16:04.183445 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:04.183410 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:04.854069 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:04.854027 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:05.470392 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:05.470346 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:06.133266 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:06.133232 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:06.772237 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:06.772201 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:07.453284 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:07.453212 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:08.067638 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:08.067602 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:08.718929 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:08.718894 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:09.362188 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:09.362131 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:10.037725 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:10.037686 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:10.694582 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:10.694553 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:11.371762 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:11.371733 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:11.974001 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:11.973972 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:12.686389 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:12.686345 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:13.364227 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:13.364193 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:14.019221 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:14.019184 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:14.677126 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:14.677087 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:15.355390 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:15.355345 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:15.962465 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:15.962430 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:16.570766 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:16.570726 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:17.253842 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:17.253808 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:17.863462 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:17.863421 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:18.613346 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:18.613302 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:19.154454 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:19.154402 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:19.867767 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:19.867724 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:20.461943 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:20.461906 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:21.193885 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:21.193849 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:21.878809 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:21.878755 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:22.556730 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:22.556675 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:23.257938 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:23.257899 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:23.878007 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:23.877969 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:24.557439 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:24.557404 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:25.180653 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:25.180606 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:25.777808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:25.777769 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:26.475294 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:26.475242 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:27.187541 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:27.187511 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:27.769759 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:27.769715 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:28.523902 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:28.523866 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:29.160345 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:29.160310 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:29.912796 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:29.912745 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:30.473710 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:30.473674 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:31.180522 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:31.180483 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:31.859325 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:31.859276 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:32.473015 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:32.472972 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:33.258721 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:33.258675 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:33.523528 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:16:33.523433 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:16:33.959414 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:33.959321 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:34.644364 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:34.644330 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:35.264393 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:35.264352 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:35.866463 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:35.866431 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:36.510022 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:36.509991 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:37.411477 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:37.411445 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:37.967578 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:37.967541 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:38.006541 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.006510 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d"] Nov 02 18:16:38.006914 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.006902 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerName="cleanup" Nov 02 18:16:38.006962 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.006916 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerName="cleanup" Nov 02 18:16:38.006962 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.006944 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerName="cleanup" Nov 02 18:16:38.006962 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.006950 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerName="cleanup" Nov 02 18:16:38.006962 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.006957 4310 cpu_manager.go:414] "containerMap: removing container" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerName="cleanup" Nov 02 18:16:38.006962 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.006962 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerName="cleanup" Nov 02 18:16:38.007163 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.007035 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerName="cleanup" Nov 02 18:16:38.007163 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.007043 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerName="cleanup" Nov 02 18:16:38.007163 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.007050 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="a667dd99-0167-4b7e-a43e-dd2583bba691" containerName="cleanup" Nov 02 18:16:38.010050 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.010034 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:38.018290 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.017697 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqh67\"" Nov 02 18:16:38.018290 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.018251 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:16:38.035122 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.035086 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d"] Nov 02 18:16:38.085543 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.085504 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:38.085733 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.085605 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lt2\" (UniqueName: \"kubernetes.io/projected/0bc0d040-57f4-48f3-950a-9860c03360ea-kube-api-access-j4lt2\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:38.085733 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.085697 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:38.186465 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.186429 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:38.186668 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.186478 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lt2\" (UniqueName: \"kubernetes.io/projected/0bc0d040-57f4-48f3-950a-9860c03360ea-kube-api-access-j4lt2\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:38.186668 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.186506 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:38.187184 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.187156 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:38.188954 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.188935 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:38.198766 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.198742 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lt2\" (UniqueName: \"kubernetes.io/projected/0bc0d040-57f4-48f3-950a-9860c03360ea-kube-api-access-j4lt2\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:38.357325 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.357234 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:38.640716 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.640692 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d"] Nov 02 18:16:38.642880 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:16:38.642851 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc0d040_57f4_48f3_950a_9860c03360ea.slice/crio-955ecb2e1db44834f0269a985ef05dd8266ad6e85ba04ea565b6f027390d2fa6 WatchSource:0}: Error finding container 955ecb2e1db44834f0269a985ef05dd8266ad6e85ba04ea565b6f027390d2fa6: Status 404 returned error can't find the container with id 955ecb2e1db44834f0269a985ef05dd8266ad6e85ba04ea565b6f027390d2fa6 Nov 02 18:16:38.850258 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:38.850202 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:39.147254 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:39.147204 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" event={"ID":"0bc0d040-57f4-48f3-950a-9860c03360ea","Type":"ContainerStarted","Data":"e7c621bfaf9b271fe98ed71e694348a7d301da3c4a3662dc13f9c6a534ac9037"} Nov 02 18:16:39.147433 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:39.147314 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" event={"ID":"0bc0d040-57f4-48f3-950a-9860c03360ea","Type":"ContainerStarted","Data":"955ecb2e1db44834f0269a985ef05dd8266ad6e85ba04ea565b6f027390d2fa6"} Nov 02 18:16:39.164170 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:39.164111 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" podStartSLOduration=2.1640715090000002 podStartE2EDuration="2.164071509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:39.163219462 +0000 UTC m=+2196.606302099" watchObservedRunningTime="2026-11-02 18:16:39.164071509 +0000 UTC m=+2196.607154145" Nov 02 18:16:39.466854 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:39.466819 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:40.065366 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:40.065310 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:40.153665 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:40.153631 4310 generic.go:356] "Generic (PLEG): container finished" podID="0bc0d040-57f4-48f3-950a-9860c03360ea" containerID="e7c621bfaf9b271fe98ed71e694348a7d301da3c4a3662dc13f9c6a534ac9037" exitCode=0 Nov 02 18:16:40.153835 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:40.153687 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" event={"ID":"0bc0d040-57f4-48f3-950a-9860c03360ea","Type":"ContainerDied","Data":"e7c621bfaf9b271fe98ed71e694348a7d301da3c4a3662dc13f9c6a534ac9037"} Nov 02 18:16:40.673423 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:40.673349 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:40.804479 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:40.804444 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:41.298254 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.298227 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:41.312711 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.312677 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-xks-ca\") pod \"0bc0d040-57f4-48f3-950a-9860c03360ea\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " Nov 02 18:16:41.312862 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.312740 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc0d040-57f4-48f3-950a-9860c03360ea-kube-api-access-j4lt2\" (UniqueName: \"kubernetes.io/projected/0bc0d040-57f4-48f3-950a-9860c03360ea-kube-api-access-j4lt2\") pod \"0bc0d040-57f4-48f3-950a-9860c03360ea\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " Nov 02 18:16:41.312862 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.312772 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-openshift-ca\") pod \"0bc0d040-57f4-48f3-950a-9860c03360ea\" (UID: \"0bc0d040-57f4-48f3-950a-9860c03360ea\") " Nov 02 18:16:41.313321 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.313286 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-openshift-ca" pod "0bc0d040-57f4-48f3-950a-9860c03360ea" (UID: "0bc0d040-57f4-48f3-950a-9860c03360ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:41.315415 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.315375 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc0d040-57f4-48f3-950a-9860c03360ea-kube-api-access-j4lt2" pod "0bc0d040-57f4-48f3-950a-9860c03360ea" (UID: "0bc0d040-57f4-48f3-950a-9860c03360ea"). InnerVolumeSpecName "kube-api-access-j4lt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:41.319380 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.319357 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-xks-ca" pod "0bc0d040-57f4-48f3-950a-9860c03360ea" (UID: "0bc0d040-57f4-48f3-950a-9860c03360ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:16:41.372264 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.372229 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:41.414502 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.414422 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:41.414502 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.414450 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc0d040-57f4-48f3-950a-9860c03360ea-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:41.414502 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.414461 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4lt2\" (UniqueName: \"kubernetes.io/projected/0bc0d040-57f4-48f3-950a-9860c03360ea-kube-api-access-j4lt2\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:41.726291 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.726256 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:41.827341 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.827282 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:41.973596 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:41.973567 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:42.164928 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:42.164850 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" Nov 02 18:16:42.165099 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:42.164850 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d" event={"ID":"0bc0d040-57f4-48f3-950a-9860c03360ea","Type":"ContainerDied","Data":"955ecb2e1db44834f0269a985ef05dd8266ad6e85ba04ea565b6f027390d2fa6"} Nov 02 18:16:42.165099 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:42.164977 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="955ecb2e1db44834f0269a985ef05dd8266ad6e85ba04ea565b6f027390d2fa6" Nov 02 18:16:42.764315 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:42.764286 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:43.470447 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:43.470411 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:44.075754 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:44.075721 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:44.676679 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:44.676646 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:45.304740 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:45.304708 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:45.956382 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:45.956342 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:46.670374 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:46.670338 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:47.371491 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:47.371461 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:48.064552 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:48.064500 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:48.128313 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:48.128279 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz"] Nov 02 18:16:48.133941 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:48.133765 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6"] Nov 02 18:16:48.196940 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:48.196910 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:48.296817 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:48.296787 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:48.715743 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:48.715716 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:49.029349 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:49.029265 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:49.129687 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:49.129649 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:53.129597 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.129519 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" podUID="f504cfbd-f914-4eb0-bd9c-fc48a881a87b" containerName="payload-processing" containerID="cri-o://fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6" gracePeriod=25 Nov 02 18:16:53.135136 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.135071 4310 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" podUID="84df1dca-0d2e-4488-af2a-2dbdb282b3c6" containerName="payload-pre-processing" containerID="cri-o://be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec" gracePeriod=25 Nov 02 18:16:53.385590 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.385517 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:16:53.423032 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.422999 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:16:53.437874 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.437845 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-kube-api-access-b7z5t\" (UniqueName: \"kubernetes.io/projected/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-kube-api-access-b7z5t\") pod \"f504cfbd-f914-4eb0-bd9c-fc48a881a87b\" (UID: \"f504cfbd-f914-4eb0-bd9c-fc48a881a87b\") " Nov 02 18:16:53.438134 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.437980 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-plugins-config-volume\") pod \"f504cfbd-f914-4eb0-bd9c-fc48a881a87b\" (UID: \"f504cfbd-f914-4eb0-bd9c-fc48a881a87b\") " Nov 02 18:16:53.438635 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.438613 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-plugins-config-volume" pod "f504cfbd-f914-4eb0-bd9c-fc48a881a87b" (UID: "f504cfbd-f914-4eb0-bd9c-fc48a881a87b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:53.441571 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.441542 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-kube-api-access-b7z5t" pod "f504cfbd-f914-4eb0-bd9c-fc48a881a87b" (UID: "f504cfbd-f914-4eb0-bd9c-fc48a881a87b"). InnerVolumeSpecName "kube-api-access-b7z5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:53.539000 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.538963 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-kube-api-access-znt69\" (UniqueName: \"kubernetes.io/projected/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-kube-api-access-znt69\") pod \"84df1dca-0d2e-4488-af2a-2dbdb282b3c6\" (UID: \"84df1dca-0d2e-4488-af2a-2dbdb282b3c6\") " Nov 02 18:16:53.539206 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.539012 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-plugins-config-volume\") pod \"84df1dca-0d2e-4488-af2a-2dbdb282b3c6\" (UID: \"84df1dca-0d2e-4488-af2a-2dbdb282b3c6\") " Nov 02 18:16:53.539394 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.539374 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:53.539446 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.539400 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7z5t\" (UniqueName: \"kubernetes.io/projected/f504cfbd-f914-4eb0-bd9c-fc48a881a87b-kube-api-access-b7z5t\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:53.539446 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.539408 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-plugins-config-volume" pod "84df1dca-0d2e-4488-af2a-2dbdb282b3c6" (UID: "84df1dca-0d2e-4488-af2a-2dbdb282b3c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:16:53.541151 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.541131 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-kube-api-access-znt69" pod "84df1dca-0d2e-4488-af2a-2dbdb282b3c6" (UID: "84df1dca-0d2e-4488-af2a-2dbdb282b3c6"). InnerVolumeSpecName "kube-api-access-znt69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:16:53.640808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.640725 4310 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-plugins-config-volume\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:53.640808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:53.640756 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znt69\" (UniqueName: \"kubernetes.io/projected/84df1dca-0d2e-4488-af2a-2dbdb282b3c6-kube-api-access-znt69\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:16:54.232240 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.232201 4310 generic.go:356] "Generic (PLEG): container finished" podID="84df1dca-0d2e-4488-af2a-2dbdb282b3c6" containerID="be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec" exitCode=0 Nov 02 18:16:54.232839 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.232291 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" event={"ID":"84df1dca-0d2e-4488-af2a-2dbdb282b3c6","Type":"ContainerDied","Data":"be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec"} Nov 02 18:16:54.233031 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.232921 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" event={"ID":"84df1dca-0d2e-4488-af2a-2dbdb282b3c6","Type":"ContainerDied","Data":"68d46c329bab903b76e5fbca27fc2fda128442ec17989d9b9f681f1beac59c38"} Nov 02 18:16:54.233031 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.232431 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6" Nov 02 18:16:54.234251 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.233581 4310 scope.go:122] "RemoveContainer" containerID="be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec" Nov 02 18:16:54.235262 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.235216 4310 generic.go:356] "Generic (PLEG): container finished" podID="f504cfbd-f914-4eb0-bd9c-fc48a881a87b" containerID="fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6" exitCode=0 Nov 02 18:16:54.235375 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.235328 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" Nov 02 18:16:54.237024 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.236998 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" event={"ID":"f504cfbd-f914-4eb0-bd9c-fc48a881a87b","Type":"ContainerDied","Data":"fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6"} Nov 02 18:16:54.237250 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.237219 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz" event={"ID":"f504cfbd-f914-4eb0-bd9c-fc48a881a87b","Type":"ContainerDied","Data":"467eb23d5f2d43178c0997190765071af307ef55c2cd4cc86177c099cb93cf94"} Nov 02 18:16:54.245098 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.245065 4310 scope.go:122] "RemoveContainer" containerID="be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec" Nov 02 18:16:54.245487 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:16:54.245467 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec\": container with ID starting with be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec not found: ID does not exist" containerID="be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec" Nov 02 18:16:54.245568 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.245496 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec"} err="failed to get container status \"be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec\": rpc error: code = NotFound desc = could not find container \"be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec\": container with ID starting with be46f82cf1bd296ed47c4894f288b849d4322cb16a0e38dbd331eb07e90c86ec not found: ID does not exist" Nov 02 18:16:54.245568 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.245514 4310 scope.go:122] "RemoveContainer" containerID="fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6" Nov 02 18:16:54.254191 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.254172 4310 scope.go:122] "RemoveContainer" containerID="fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6" Nov 02 18:16:54.254488 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:16:54.254471 4310 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6\": container with ID starting with fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6 not found: ID does not exist" containerID="fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6" Nov 02 18:16:54.254550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.254500 4310 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6"} err="failed to get container status \"fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6\": rpc error: code = NotFound desc = could not find container \"fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6\": container with ID starting with fe3e809e0c722822459b7e60e8a9905dc9813fcf6d3d3b5dcc223604131675d6 not found: ID does not exist" Nov 02 18:16:54.258752 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.258729 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:54.260520 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.260497 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz"] Nov 02 18:16:54.263506 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.263485 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5d4d8d-5659fccfdf-qqbpz"] Nov 02 18:16:54.274581 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.274553 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6"] Nov 02 18:16:54.280477 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.280453 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5d4d8d-7c6d556b4c-n9mp6"] Nov 02 18:16:54.406444 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.406357 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:54.441573 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.441538 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d"] Nov 02 18:16:54.446984 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.446960 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5d4d8d-bgk2d"] Nov 02 18:16:54.504257 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.504210 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:16:54.672323 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.672239 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc0d040-57f4-48f3-950a-9860c03360ea" path="/var/lib/kubelet/pods/0bc0d040-57f4-48f3-950a-9860c03360ea/volumes" Nov 02 18:16:54.672628 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.672614 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84df1dca-0d2e-4488-af2a-2dbdb282b3c6" path="/var/lib/kubelet/pods/84df1dca-0d2e-4488-af2a-2dbdb282b3c6/volumes" Nov 02 18:16:54.672962 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.672933 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f504cfbd-f914-4eb0-bd9c-fc48a881a87b" path="/var/lib/kubelet/pods/f504cfbd-f914-4eb0-bd9c-fc48a881a87b/volumes" Nov 02 18:16:54.973109 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:16:54.973058 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:00.753944 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:00.753903 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:00.854608 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:00.854577 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:01.133789 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:01.133704 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:03.524582 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:17:03.524539 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:17:13.040342 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.040300 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v"] Nov 02 18:17:13.040777 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.040698 4310 cpu_manager.go:414] "containerMap: removing container" podUID="84df1dca-0d2e-4488-af2a-2dbdb282b3c6" containerName="payload-pre-processing" Nov 02 18:17:13.040777 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.040710 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84df1dca-0d2e-4488-af2a-2dbdb282b3c6" containerName="payload-pre-processing" Nov 02 18:17:13.040777 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.040741 4310 cpu_manager.go:414] "containerMap: removing container" podUID="0bc0d040-57f4-48f3-950a-9860c03360ea" containerName="revoke-keys" Nov 02 18:17:13.040777 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.040747 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc0d040-57f4-48f3-950a-9860c03360ea" containerName="revoke-keys" Nov 02 18:17:13.040777 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.040762 4310 cpu_manager.go:414] "containerMap: removing container" podUID="f504cfbd-f914-4eb0-bd9c-fc48a881a87b" containerName="payload-processing" Nov 02 18:17:13.040777 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.040770 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f504cfbd-f914-4eb0-bd9c-fc48a881a87b" containerName="payload-processing" Nov 02 18:17:13.041012 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.040842 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="84df1dca-0d2e-4488-af2a-2dbdb282b3c6" containerName="payload-pre-processing" Nov 02 18:17:13.041012 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.040854 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc0d040-57f4-48f3-950a-9860c03360ea" containerName="revoke-keys" Nov 02 18:17:13.041012 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.040862 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="f504cfbd-f914-4eb0-bd9c-fc48a881a87b" containerName="payload-processing" Nov 02 18:17:13.044870 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.044851 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:13.049357 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.049336 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:17:13.049465 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.049444 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqh67\"" Nov 02 18:17:13.052464 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.052437 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v"] Nov 02 18:17:13.210025 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.209993 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wcv8v\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:13.210242 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.210053 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wcv8v\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:13.210242 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.210188 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v624d\" (UniqueName: \"kubernetes.io/projected/ad163654-5fac-411d-811b-e1628c0f0384-kube-api-access-v624d\") pod \"maas-api-revoke-sub-f03629db-wcv8v\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:13.311101 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.310981 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wcv8v\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:13.311101 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.311061 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wcv8v\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:13.311294 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.311130 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v624d\" (UniqueName: \"kubernetes.io/projected/ad163654-5fac-411d-811b-e1628c0f0384-kube-api-access-v624d\") pod \"maas-api-revoke-sub-f03629db-wcv8v\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:13.311828 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.311808 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wcv8v\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:13.313348 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.313320 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wcv8v\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:13.319811 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.319789 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v624d\" (UniqueName: \"kubernetes.io/projected/ad163654-5fac-411d-811b-e1628c0f0384-kube-api-access-v624d\") pod \"maas-api-revoke-sub-f03629db-wcv8v\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:13.360497 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.360466 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:13.632926 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:13.632900 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v"] Nov 02 18:17:13.634733 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:17:13.634699 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad163654_5fac_411d_811b_e1628c0f0384.slice/crio-f19ff4ad12cc361cc5334b8f1478f1e9575b49c6bf6dd7983060eb8c1e48329c WatchSource:0}: Error finding container f19ff4ad12cc361cc5334b8f1478f1e9575b49c6bf6dd7983060eb8c1e48329c: Status 404 returned error can't find the container with id f19ff4ad12cc361cc5334b8f1478f1e9575b49c6bf6dd7983060eb8c1e48329c Nov 02 18:17:14.019922 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:14.019886 4310 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj"] Nov 02 18:17:14.022261 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:14.022233 4310 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ce9c90-nx5tj"] Nov 02 18:17:14.351541 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:14.351508 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" event={"ID":"ad163654-5fac-411d-811b-e1628c0f0384","Type":"ContainerStarted","Data":"e7e082e7dea6fa5396e37c6ba7e48972194a1daf608457cdd07a6c225ca198a2"} Nov 02 18:17:14.351998 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:14.351558 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" event={"ID":"ad163654-5fac-411d-811b-e1628c0f0384","Type":"ContainerStarted","Data":"f19ff4ad12cc361cc5334b8f1478f1e9575b49c6bf6dd7983060eb8c1e48329c"} Nov 02 18:17:14.368262 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:14.368209 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" podStartSLOduration=1.368195695 podStartE2EDuration="1.368195695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:14.366958705 +0000 UTC m=+2231.810041341" watchObservedRunningTime="2026-11-02 18:17:14.368195695 +0000 UTC m=+2231.811278330" Nov 02 18:17:14.673036 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:14.672952 4310 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5baccd4-9912-470d-af3e-ada2b40c3e20" path="/var/lib/kubelet/pods/c5baccd4-9912-470d-af3e-ada2b40c3e20/volumes" Nov 02 18:17:15.357033 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:15.356998 4310 generic.go:356] "Generic (PLEG): container finished" podID="ad163654-5fac-411d-811b-e1628c0f0384" containerID="e7e082e7dea6fa5396e37c6ba7e48972194a1daf608457cdd07a6c225ca198a2" exitCode=0 Nov 02 18:17:15.357458 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:15.357044 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" event={"ID":"ad163654-5fac-411d-811b-e1628c0f0384","Type":"ContainerDied","Data":"e7e082e7dea6fa5396e37c6ba7e48972194a1daf608457cdd07a6c225ca198a2"} Nov 02 18:17:16.490448 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:16.490420 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:16.648042 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:16.647951 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad163654-5fac-411d-811b-e1628c0f0384-kube-api-access-v624d\" (UniqueName: \"kubernetes.io/projected/ad163654-5fac-411d-811b-e1628c0f0384-kube-api-access-v624d\") pod \"ad163654-5fac-411d-811b-e1628c0f0384\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " Nov 02 18:17:16.648042 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:16.648039 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-xks-ca\") pod \"ad163654-5fac-411d-811b-e1628c0f0384\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " Nov 02 18:17:16.648276 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:16.648162 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-openshift-ca\") pod \"ad163654-5fac-411d-811b-e1628c0f0384\" (UID: \"ad163654-5fac-411d-811b-e1628c0f0384\") " Nov 02 18:17:16.648580 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:16.648554 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-openshift-ca" pod "ad163654-5fac-411d-811b-e1628c0f0384" (UID: "ad163654-5fac-411d-811b-e1628c0f0384"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:17:16.650246 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:16.650223 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-xks-ca" pod "ad163654-5fac-411d-811b-e1628c0f0384" (UID: "ad163654-5fac-411d-811b-e1628c0f0384"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:17:16.650337 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:16.650277 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad163654-5fac-411d-811b-e1628c0f0384-kube-api-access-v624d" pod "ad163654-5fac-411d-811b-e1628c0f0384" (UID: "ad163654-5fac-411d-811b-e1628c0f0384"). InnerVolumeSpecName "kube-api-access-v624d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:17:16.749915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:16.749879 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v624d\" (UniqueName: \"kubernetes.io/projected/ad163654-5fac-411d-811b-e1628c0f0384-kube-api-access-v624d\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:17:16.749915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:16.749909 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:17:16.749915 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:16.749919 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad163654-5fac-411d-811b-e1628c0f0384-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:17:17.366734 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:17.366700 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" Nov 02 18:17:17.366924 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:17.366732 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wcv8v" event={"ID":"ad163654-5fac-411d-811b-e1628c0f0384","Type":"ContainerDied","Data":"f19ff4ad12cc361cc5334b8f1478f1e9575b49c6bf6dd7983060eb8c1e48329c"} Nov 02 18:17:17.366924 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:17.366788 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19ff4ad12cc361cc5334b8f1478f1e9575b49c6bf6dd7983060eb8c1e48329c" Nov 02 18:17:17.907285 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:17.907255 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:17:29.366827 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.366790 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc"] Nov 02 18:17:29.367489 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.367263 4310 cpu_manager.go:414] "containerMap: removing container" podUID="ad163654-5fac-411d-811b-e1628c0f0384" containerName="revoke-keys" Nov 02 18:17:29.367489 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.367276 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad163654-5fac-411d-811b-e1628c0f0384" containerName="revoke-keys" Nov 02 18:17:29.367489 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.367361 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad163654-5fac-411d-811b-e1628c0f0384" containerName="revoke-keys" Nov 02 18:17:29.370456 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.370435 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:29.377373 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.377344 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc"] Nov 02 18:17:29.377702 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.377679 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqh67\"" Nov 02 18:17:29.377855 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.377723 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:17:29.499180 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.499144 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-k49fc\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:29.499365 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.499279 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjznz\" (UniqueName: \"kubernetes.io/projected/51ca605c-f939-4071-8e8a-2bac02c53ea1-kube-api-access-fjznz\") pod \"maas-api-revoke-sub-0165d7d2-k49fc\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:29.499365 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.499326 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-k49fc\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:29.600094 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.600049 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-k49fc\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:29.600283 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.600209 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjznz\" (UniqueName: \"kubernetes.io/projected/51ca605c-f939-4071-8e8a-2bac02c53ea1-kube-api-access-fjznz\") pod \"maas-api-revoke-sub-0165d7d2-k49fc\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:29.600283 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.600256 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-k49fc\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:29.600986 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.600945 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-k49fc\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:29.602476 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.602455 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-k49fc\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:29.608460 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.608441 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjznz\" (UniqueName: \"kubernetes.io/projected/51ca605c-f939-4071-8e8a-2bac02c53ea1-kube-api-access-fjznz\") pod \"maas-api-revoke-sub-0165d7d2-k49fc\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:29.690996 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.690909 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:29.963663 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:29.963634 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc"] Nov 02 18:17:29.967024 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:17:29.966994 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ca605c_f939_4071_8e8a_2bac02c53ea1.slice/crio-cb011076c4eb1af731c43d07f24643e5ae11ec59274a185672d44c3952e9a79c WatchSource:0}: Error finding container cb011076c4eb1af731c43d07f24643e5ae11ec59274a185672d44c3952e9a79c: Status 404 returned error can't find the container with id cb011076c4eb1af731c43d07f24643e5ae11ec59274a185672d44c3952e9a79c Nov 02 18:17:30.440208 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:30.440168 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" event={"ID":"51ca605c-f939-4071-8e8a-2bac02c53ea1","Type":"ContainerStarted","Data":"0e20251b92de7d1d11aec0fe3f0fa6b9d8b65b0bd12b5d6d25c4c323080703d6"} Nov 02 18:17:30.440696 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:30.440232 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" event={"ID":"51ca605c-f939-4071-8e8a-2bac02c53ea1","Type":"ContainerStarted","Data":"cb011076c4eb1af731c43d07f24643e5ae11ec59274a185672d44c3952e9a79c"} Nov 02 18:17:30.456986 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:30.456919 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" podStartSLOduration=1.45690128 podStartE2EDuration="1.45690128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:30.456602049 +0000 UTC m=+2247.899684688" watchObservedRunningTime="2026-11-02 18:17:30.45690128 +0000 UTC m=+2247.899984015" Nov 02 18:17:31.445684 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:31.445645 4310 generic.go:356] "Generic (PLEG): container finished" podID="51ca605c-f939-4071-8e8a-2bac02c53ea1" containerID="0e20251b92de7d1d11aec0fe3f0fa6b9d8b65b0bd12b5d6d25c4c323080703d6" exitCode=0 Nov 02 18:17:31.446134 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:31.445731 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" event={"ID":"51ca605c-f939-4071-8e8a-2bac02c53ea1","Type":"ContainerDied","Data":"0e20251b92de7d1d11aec0fe3f0fa6b9d8b65b0bd12b5d6d25c4c323080703d6"} Nov 02 18:17:32.590608 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:32.590582 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:32.731348 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:32.731315 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-openshift-ca\") pod \"51ca605c-f939-4071-8e8a-2bac02c53ea1\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " Nov 02 18:17:32.731528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:32.731405 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-xks-ca\") pod \"51ca605c-f939-4071-8e8a-2bac02c53ea1\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " Nov 02 18:17:32.731528 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:32.731434 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ca605c-f939-4071-8e8a-2bac02c53ea1-kube-api-access-fjznz\" (UniqueName: \"kubernetes.io/projected/51ca605c-f939-4071-8e8a-2bac02c53ea1-kube-api-access-fjznz\") pod \"51ca605c-f939-4071-8e8a-2bac02c53ea1\" (UID: \"51ca605c-f939-4071-8e8a-2bac02c53ea1\") " Nov 02 18:17:32.731701 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:32.731675 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-openshift-ca" pod "51ca605c-f939-4071-8e8a-2bac02c53ea1" (UID: "51ca605c-f939-4071-8e8a-2bac02c53ea1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:17:32.731809 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:32.731780 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:17:32.733491 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:32.733466 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-xks-ca" pod "51ca605c-f939-4071-8e8a-2bac02c53ea1" (UID: "51ca605c-f939-4071-8e8a-2bac02c53ea1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:17:32.733584 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:32.733540 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ca605c-f939-4071-8e8a-2bac02c53ea1-kube-api-access-fjznz" pod "51ca605c-f939-4071-8e8a-2bac02c53ea1" (UID: "51ca605c-f939-4071-8e8a-2bac02c53ea1"). InnerVolumeSpecName "kube-api-access-fjznz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:17:32.832889 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:32.832862 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ca605c-f939-4071-8e8a-2bac02c53ea1-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:17:32.832889 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:32.832887 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjznz\" (UniqueName: \"kubernetes.io/projected/51ca605c-f939-4071-8e8a-2bac02c53ea1-kube-api-access-fjznz\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:17:33.456599 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:33.456569 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" Nov 02 18:17:33.456787 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:33.456611 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-k49fc" event={"ID":"51ca605c-f939-4071-8e8a-2bac02c53ea1","Type":"ContainerDied","Data":"cb011076c4eb1af731c43d07f24643e5ae11ec59274a185672d44c3952e9a79c"} Nov 02 18:17:33.456787 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:33.456680 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb011076c4eb1af731c43d07f24643e5ae11ec59274a185672d44c3952e9a79c" Nov 02 18:17:33.522222 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:17:33.522183 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:17:40.157020 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.156975 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465"] Nov 02 18:17:40.157654 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.157627 4310 cpu_manager.go:414] "containerMap: removing container" podUID="51ca605c-f939-4071-8e8a-2bac02c53ea1" containerName="revoke-keys" Nov 02 18:17:40.157654 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.157654 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ca605c-f939-4071-8e8a-2bac02c53ea1" containerName="revoke-keys" Nov 02 18:17:40.157857 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.157794 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ca605c-f939-4071-8e8a-2bac02c53ea1" containerName="revoke-keys" Nov 02 18:17:40.161443 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.161421 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:40.167048 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.167009 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465"] Nov 02 18:17:40.192094 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.192057 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:17:40.192237 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.192122 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqh67\"" Nov 02 18:17:40.307943 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.307907 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gr465\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:40.308148 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.308038 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gr465\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:40.308148 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.308110 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9t67\" (UniqueName: \"kubernetes.io/projected/84e01e7d-9d28-4f94-a38d-a37efe698589-kube-api-access-h9t67\") pod \"maas-api-revoke-sub-6b19338f-gr465\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:40.408710 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.408617 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gr465\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:40.408710 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.408710 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gr465\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:40.408914 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.408742 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9t67\" (UniqueName: \"kubernetes.io/projected/84e01e7d-9d28-4f94-a38d-a37efe698589-kube-api-access-h9t67\") pod \"maas-api-revoke-sub-6b19338f-gr465\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:40.409316 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.409296 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gr465\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:40.411101 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.411073 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gr465\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:40.416490 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.416467 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9t67\" (UniqueName: \"kubernetes.io/projected/84e01e7d-9d28-4f94-a38d-a37efe698589-kube-api-access-h9t67\") pod \"maas-api-revoke-sub-6b19338f-gr465\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:40.477067 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.477029 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:40.782604 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:40.782578 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465"] Nov 02 18:17:40.784766 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:17:40.784740 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e01e7d_9d28_4f94_a38d_a37efe698589.slice/crio-75fbd7055842cd50e76600de3416301747c16e841f71f76e3c831ba71758eab5 WatchSource:0}: Error finding container 75fbd7055842cd50e76600de3416301747c16e841f71f76e3c831ba71758eab5: Status 404 returned error can't find the container with id 75fbd7055842cd50e76600de3416301747c16e841f71f76e3c831ba71758eab5 Nov 02 18:17:41.498544 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:41.498502 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" event={"ID":"84e01e7d-9d28-4f94-a38d-a37efe698589","Type":"ContainerStarted","Data":"977f817e8cd3dcfb1e8ad3357703c6698e280af0d1063a03c5374bba37c536cb"} Nov 02 18:17:41.498959 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:41.498569 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" event={"ID":"84e01e7d-9d28-4f94-a38d-a37efe698589","Type":"ContainerStarted","Data":"75fbd7055842cd50e76600de3416301747c16e841f71f76e3c831ba71758eab5"} Nov 02 18:17:41.516318 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:41.516264 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" podStartSLOduration=1.516248204 podStartE2EDuration="1.516248204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:41.51483745 +0000 UTC m=+2258.957920100" watchObservedRunningTime="2026-11-02 18:17:41.516248204 +0000 UTC m=+2258.959330870" Nov 02 18:17:42.510240 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:42.510200 4310 generic.go:356] "Generic (PLEG): container finished" podID="84e01e7d-9d28-4f94-a38d-a37efe698589" containerID="977f817e8cd3dcfb1e8ad3357703c6698e280af0d1063a03c5374bba37c536cb" exitCode=0 Nov 02 18:17:42.510639 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:42.510262 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" event={"ID":"84e01e7d-9d28-4f94-a38d-a37efe698589","Type":"ContainerDied","Data":"977f817e8cd3dcfb1e8ad3357703c6698e280af0d1063a03c5374bba37c536cb"} Nov 02 18:17:43.648424 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:43.648398 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:43.739860 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:43.739821 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e01e7d-9d28-4f94-a38d-a37efe698589-kube-api-access-h9t67\" (UniqueName: \"kubernetes.io/projected/84e01e7d-9d28-4f94-a38d-a37efe698589-kube-api-access-h9t67\") pod \"84e01e7d-9d28-4f94-a38d-a37efe698589\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " Nov 02 18:17:43.740072 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:43.739945 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-xks-ca\") pod \"84e01e7d-9d28-4f94-a38d-a37efe698589\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " Nov 02 18:17:43.740072 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:43.739980 4310 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-openshift-ca\") pod \"84e01e7d-9d28-4f94-a38d-a37efe698589\" (UID: \"84e01e7d-9d28-4f94-a38d-a37efe698589\") " Nov 02 18:17:43.740448 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:43.740421 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-openshift-ca" pod "84e01e7d-9d28-4f94-a38d-a37efe698589" (UID: "84e01e7d-9d28-4f94-a38d-a37efe698589"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:17:43.742152 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:43.742118 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-xks-ca" pod "84e01e7d-9d28-4f94-a38d-a37efe698589" (UID: "84e01e7d-9d28-4f94-a38d-a37efe698589"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:17:43.742262 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:43.742220 4310 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e01e7d-9d28-4f94-a38d-a37efe698589-kube-api-access-h9t67" pod "84e01e7d-9d28-4f94-a38d-a37efe698589" (UID: "84e01e7d-9d28-4f94-a38d-a37efe698589"). InnerVolumeSpecName "kube-api-access-h9t67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:17:43.840803 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:43.840714 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-xks-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:17:43.840803 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:43.840750 4310 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e01e7d-9d28-4f94-a38d-a37efe698589-maas-api-openshift-ca\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:17:43.840803 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:43.840760 4310 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9t67\" (UniqueName: \"kubernetes.io/projected/84e01e7d-9d28-4f94-a38d-a37efe698589-kube-api-access-h9t67\") on node \"ip-10-0-135-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 18:17:44.521692 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:44.521661 4310 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" Nov 02 18:17:44.521878 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:44.521660 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gr465" event={"ID":"84e01e7d-9d28-4f94-a38d-a37efe698589","Type":"ContainerDied","Data":"75fbd7055842cd50e76600de3416301747c16e841f71f76e3c831ba71758eab5"} Nov 02 18:17:44.521878 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:44.521796 4310 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75fbd7055842cd50e76600de3416301747c16e841f71f76e3c831ba71758eab5" Nov 02 18:17:49.184941 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:17:49.184901 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:03.523192 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:18:03.523147 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:18:11.493808 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:11.493776 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-fdd7ff9-r2xj6_325e80f9-6751-44b2-a3b2-0c2b1d2e4def/manager/0.log" Nov 02 18:18:11.683924 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:11.683885 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4pwjv_05242e7d-02d8-4b62-a200-cdd9f108e64a/manager/0.log" Nov 02 18:18:11.842456 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:11.842372 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rxjvx_af9c459a-3726-41aa-b94d-02bdcfd9d10e/manager/0.log" Nov 02 18:18:12.309974 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:12.309939 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-698c564c8b-rgrcd_6058e7f5-08e4-47e6-a656-9fc94d877250/manager/0.log" Nov 02 18:18:12.532440 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:12.532415 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5x726_94716789-2a04-4aa1-a10e-b6e2fb1517e5/server/0.log" Nov 02 18:18:12.706643 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:12.706563 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5ncdw_abdb4611-dc86-47e3-997a-afb58e26fade/manager/0.log" Nov 02 18:18:12.906487 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:12.906446 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-8twlt_1705d3f8-7617-44ed-b8c5-6adb2e14cb55/proxy/0.log" Nov 02 18:18:13.979018 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:13.978987 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-k49fc_51ca605c-f939-4071-8e8a-2bac02c53ea1/revoke-keys/0.log" Nov 02 18:18:14.642324 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:14.642288 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:14.672586 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:14.672555 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-gr465_84e01e7d-9d28-4f94-a38d-a37efe698589/revoke-keys/0.log" Nov 02 18:18:14.844324 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:14.844299 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-wcv8v_ad163654-5fac-411d-811b-e1628c0f0384/revoke-keys/0.log" Nov 02 18:18:15.014866 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:15.014833 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-g4jn7_1956a285-a1d9-4dd6-aa86-fa73ec4272fc/postgres/0.log" Nov 02 18:18:18.511922 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:18.511887 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8f8hh_26d61838-6558-489f-a369-b8dd6afe14f5/discovery/0.log" Nov 02 18:18:18.710955 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:18.710927 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555d4cc554-ssg7x_4de65c0e-62d2-4e29-b13f-309a0ce43deb/kube-auth-proxy/0.log" Nov 02 18:18:19.544239 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:19.544208 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8ftbd_8d9f41e4-d435-4f1f-95dc-858820a331bf/istio-proxy/0.log" Nov 02 18:18:20.055422 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:20.055393 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-l98w5_d1be36c1-1e0e-478d-bdeb-7a9704cee2e0/istio-proxy/0.log" Nov 02 18:18:20.571630 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:20.571606 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-vkwgh_1a783ce1-3916-4723-a400-9155ea747d15/istio-proxy/0.log" Nov 02 18:18:22.351334 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:22.351302 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-jdv66_2144453b-de91-4404-8b54-5960b5a697ce/main/0.log" Nov 02 18:18:22.539319 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:22.539287 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-4v5sg_3b41c5ba-f619-4894-8e4f-427014637518/main/0.log" Nov 02 18:18:22.871654 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:22.871621 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp_d4955081-1a71-4933-8332-dd8bc035ae18/main/0.log" Nov 02 18:18:23.041749 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:23.041721 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-6jcls_c19cdb66-3f5e-4ac8-a77c-d0a85d130613/main/0.log" Nov 02 18:18:32.864608 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:32.864578 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-fdd7ff9-r2xj6_325e80f9-6751-44b2-a3b2-0c2b1d2e4def/manager/0.log" Nov 02 18:18:33.059569 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:33.059535 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4pwjv_05242e7d-02d8-4b62-a200-cdd9f108e64a/manager/0.log" Nov 02 18:18:33.218912 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:33.218885 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rxjvx_af9c459a-3726-41aa-b94d-02bdcfd9d10e/manager/0.log" Nov 02 18:18:33.522468 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:18:33.522387 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:18:33.634491 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:33.634460 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-698c564c8b-rgrcd_6058e7f5-08e4-47e6-a656-9fc94d877250/manager/0.log" Nov 02 18:18:33.841619 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:33.841537 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5x726_94716789-2a04-4aa1-a10e-b6e2fb1517e5/server/0.log" Nov 02 18:18:34.009301 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:34.009270 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5ncdw_abdb4611-dc86-47e3-997a-afb58e26fade/manager/0.log" Nov 02 18:18:34.201012 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:34.200936 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-8twlt_1705d3f8-7617-44ed-b8c5-6adb2e14cb55/proxy/0.log" Nov 02 18:18:35.025430 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:35.025399 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-k49fc_51ca605c-f939-4071-8e8a-2bac02c53ea1/revoke-keys/0.log" Nov 02 18:18:35.540589 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:35.540561 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-gr465_84e01e7d-9d28-4f94-a38d-a37efe698589/revoke-keys/0.log" Nov 02 18:18:35.702959 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:35.702934 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-wcv8v_ad163654-5fac-411d-811b-e1628c0f0384/revoke-keys/0.log" Nov 02 18:18:35.866266 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:35.866186 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-g4jn7_1956a285-a1d9-4dd6-aa86-fa73ec4272fc/postgres/0.log" Nov 02 18:18:48.852007 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:48.851977 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8f8hh_26d61838-6558-489f-a369-b8dd6afe14f5/discovery/0.log" Nov 02 18:18:49.027614 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:49.027581 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555d4cc554-ssg7x_4de65c0e-62d2-4e29-b13f-309a0ce43deb/kube-auth-proxy/0.log" Nov 02 18:18:49.128465 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:49.128368 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:18:49.878313 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:49.878279 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8ftbd_8d9f41e4-d435-4f1f-95dc-858820a331bf/istio-proxy/0.log" Nov 02 18:18:50.398110 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:18:50.398062 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-l98w5_d1be36c1-1e0e-478d-bdeb-7a9704cee2e0/istio-proxy/0.log" Nov 02 18:19:00.925212 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:19:00.925175 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-vkwgh_1a783ce1-3916-4723-a400-9155ea747d15/istio-proxy/0.log" Nov 02 18:19:02.309728 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:19:02.309700 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-jdv66_2144453b-de91-4404-8b54-5960b5a697ce/main/0.log" Nov 02 18:19:02.478442 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:19:02.478413 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-4v5sg_3b41c5ba-f619-4894-8e4f-427014637518/main/0.log" Nov 02 18:19:02.814171 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:19:02.814143 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hdnvp_d4955081-1a71-4933-8332-dd8bc035ae18/main/0.log" Nov 02 18:19:02.999237 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:19:02.999205 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-6jcls_c19cdb66-3f5e-4ac8-a77c-d0a85d130613/main/0.log" Nov 02 18:19:03.522526 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:19:03.522492 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:19:05.742191 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:19:05.742157 4310 scope.go:122] "RemoveContainer" containerID="0c34bec27ef9fcc5e0c806f2c279e37251c23a3b07be15a3da01ea173993a270" Nov 02 18:19:05.755942 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:19:05.755903 4310 scope.go:122] "RemoveContainer" containerID="b75c44e44e547f7daa082c39b809cd7fa2d9eaa0675893a1f7394ac4239e3a60" Nov 02 18:19:33.522648 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:19:33.522610 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:19:49.162367 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:19:49.162285 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:19:49.428585 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:19:49.428498 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-698c564c8b-rgrcd_6058e7f5-08e4-47e6-a656-9fc94d877250/manager/0.log" Nov 02 18:19:49.685816 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:19:49.685727 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-698c564c8b-rgrcd_6058e7f5-08e4-47e6-a656-9fc94d877250/manager/0.log" Nov 02 18:20:03.522992 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:20:03.522949 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:20:30.150206 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:30.150171 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:33.523626 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:20:33.523589 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:20:34.114706 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:34.114670 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-698c564c8b-rgrcd_6058e7f5-08e4-47e6-a656-9fc94d877250/manager/0.log" Nov 02 18:20:34.369634 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:34.369555 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-698c564c8b-rgrcd_6058e7f5-08e4-47e6-a656-9fc94d877250/manager/0.log" Nov 02 18:20:35.475386 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.475350 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:35.493395 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.493355 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4fmsl/must-gather-zqtk6"] Nov 02 18:20:35.493836 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.493821 4310 cpu_manager.go:414] "containerMap: removing container" podUID="84e01e7d-9d28-4f94-a38d-a37efe698589" containerName="revoke-keys" Nov 02 18:20:35.493886 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.493838 4310 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e01e7d-9d28-4f94-a38d-a37efe698589" containerName="revoke-keys" Nov 02 18:20:35.493923 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.493914 4310 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e01e7d-9d28-4f94-a38d-a37efe698589" containerName="revoke-keys" Nov 02 18:20:35.497220 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.497201 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fmsl/must-gather-zqtk6" Nov 02 18:20:35.501024 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.500993 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4fmsl\"/\"openshift-service-ca.crt\"" Nov 02 18:20:35.502050 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.502028 4310 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4fmsl\"/\"kube-root-ca.crt\"" Nov 02 18:20:35.502050 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.502033 4310 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4fmsl\"/\"default-dockercfg-md9st\"" Nov 02 18:20:35.514590 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.514560 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4fmsl/must-gather-zqtk6"] Nov 02 18:20:35.568662 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.568625 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:35.689278 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.689242 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxc44\" (UniqueName: \"kubernetes.io/projected/998e950a-bddc-4547-87fc-1dee19727f54-kube-api-access-zxc44\") pod \"must-gather-zqtk6\" (UID: \"998e950a-bddc-4547-87fc-1dee19727f54\") " pod="openshift-must-gather-4fmsl/must-gather-zqtk6" Nov 02 18:20:35.689494 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.689310 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/998e950a-bddc-4547-87fc-1dee19727f54-must-gather-output\") pod \"must-gather-zqtk6\" (UID: \"998e950a-bddc-4547-87fc-1dee19727f54\") " pod="openshift-must-gather-4fmsl/must-gather-zqtk6" Nov 02 18:20:35.790304 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.790220 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxc44\" (UniqueName: \"kubernetes.io/projected/998e950a-bddc-4547-87fc-1dee19727f54-kube-api-access-zxc44\") pod \"must-gather-zqtk6\" (UID: \"998e950a-bddc-4547-87fc-1dee19727f54\") " pod="openshift-must-gather-4fmsl/must-gather-zqtk6" Nov 02 18:20:35.790304 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.790278 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/998e950a-bddc-4547-87fc-1dee19727f54-must-gather-output\") pod \"must-gather-zqtk6\" (UID: \"998e950a-bddc-4547-87fc-1dee19727f54\") " pod="openshift-must-gather-4fmsl/must-gather-zqtk6" Nov 02 18:20:35.790595 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.790580 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/998e950a-bddc-4547-87fc-1dee19727f54-must-gather-output\") pod \"must-gather-zqtk6\" (UID: \"998e950a-bddc-4547-87fc-1dee19727f54\") " pod="openshift-must-gather-4fmsl/must-gather-zqtk6" Nov 02 18:20:35.800309 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.800283 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxc44\" (UniqueName: \"kubernetes.io/projected/998e950a-bddc-4547-87fc-1dee19727f54-kube-api-access-zxc44\") pod \"must-gather-zqtk6\" (UID: \"998e950a-bddc-4547-87fc-1dee19727f54\") " pod="openshift-must-gather-4fmsl/must-gather-zqtk6" Nov 02 18:20:35.827398 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:35.827362 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fmsl/must-gather-zqtk6" Nov 02 18:20:36.101261 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:36.101188 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4fmsl/must-gather-zqtk6"] Nov 02 18:20:36.103797 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:20:36.103763 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod998e950a_bddc_4547_87fc_1dee19727f54.slice/crio-1c7a544766ed1fc31a08adcdfa319c619c056ec6d158ba22bc4c44c9f3723dae WatchSource:0}: Error finding container 1c7a544766ed1fc31a08adcdfa319c619c056ec6d158ba22bc4c44c9f3723dae: Status 404 returned error can't find the container with id 1c7a544766ed1fc31a08adcdfa319c619c056ec6d158ba22bc4c44c9f3723dae Nov 02 18:20:36.106028 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:36.106010 4310 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:20:36.362581 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:36.362497 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fmsl/must-gather-zqtk6" event={"ID":"998e950a-bddc-4547-87fc-1dee19727f54","Type":"ContainerStarted","Data":"1c7a544766ed1fc31a08adcdfa319c619c056ec6d158ba22bc4c44c9f3723dae"} Nov 02 18:20:37.369865 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:37.369832 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fmsl/must-gather-zqtk6" event={"ID":"998e950a-bddc-4547-87fc-1dee19727f54","Type":"ContainerStarted","Data":"4e54956c5abc7f7fcc076b0cff6be5ea21d577016b3617e3f996781b1291de47"} Nov 02 18:20:37.370282 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:37.369874 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fmsl/must-gather-zqtk6" event={"ID":"998e950a-bddc-4547-87fc-1dee19727f54","Type":"ContainerStarted","Data":"40487734f3196b741ebc46053893fd24418923a2985757de0f153de241c94b9b"} Nov 02 18:20:37.387672 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:37.387618 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4fmsl/must-gather-zqtk6" podStartSLOduration=1.603899263 podStartE2EDuration="2.387601888s" totalImagesPullingTime="783.702625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:37.385122163 +0000 UTC m=+2434.828204796" watchObservedRunningTime="2026-11-02 18:20:37.387601888 +0000 UTC m=+2434.830684524" Nov 02 18:20:38.626904 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:38.626874 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pbcbk_ea8500af-fe54-4c1e-983d-7a7b82d866b3/global-pull-secret-syncer/0.log" Nov 02 18:20:38.872295 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:38.872259 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" Nov 02 18:20:38.985383 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:38.985347 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-9.us-east-2.compute.internal_06cff5a60959fbf288446179900e7d21/haproxy/0.log" Nov 02 18:20:47.661102 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:47.658829 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bfccc956c-622tf_6dfe31f7-f071-4aac-aa92-ae6509305835/metrics-server/0.log" Nov 02 18:20:47.808102 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:47.807538 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-85vm8_6ac8dbad-7669-486e-9580-580b89796ba0/node-exporter/0.log" Nov 02 18:20:47.834472 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:47.834438 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-85vm8_6ac8dbad-7669-486e-9580-580b89796ba0/kube-rbac-proxy/0.log" Nov 02 18:20:47.856238 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:47.856211 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-85vm8_6ac8dbad-7669-486e-9580-580b89796ba0/init-textfile/0.log" Nov 02 18:20:48.258032 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.258003 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-bvzg2_87d8bb05-337f-4eb5-b71f-f95f3b160ecb/prometheus-operator-admission-webhook/0.log" Nov 02 18:20:48.380320 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.380289 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c47fdcc77-94c69_54365f2d-6ca2-4f59-8c06-c5406f9857e3/thanos-query/0.log" Nov 02 18:20:48.402391 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.402355 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c47fdcc77-94c69_54365f2d-6ca2-4f59-8c06-c5406f9857e3/kube-rbac-proxy-web/0.log" Nov 02 18:20:48.425499 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.425445 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c47fdcc77-94c69_54365f2d-6ca2-4f59-8c06-c5406f9857e3/kube-rbac-proxy/0.log" Nov 02 18:20:48.448136 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.447643 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c47fdcc77-94c69_54365f2d-6ca2-4f59-8c06-c5406f9857e3/prom-label-proxy/0.log" Nov 02 18:20:48.477474 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.477445 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c47fdcc77-94c69_54365f2d-6ca2-4f59-8c06-c5406f9857e3/kube-rbac-proxy-rules/0.log" Nov 02 18:20:48.499422 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.499390 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c47fdcc77-94c69_54365f2d-6ca2-4f59-8c06-c5406f9857e3/kube-rbac-proxy-metrics/0.log" Nov 02 18:20:48.580875 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.580554 4310 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq"] Nov 02 18:20:48.590522 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.590470 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.594793 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.594765 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq"] Nov 02 18:20:48.672301 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.672266 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-proc\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.672799 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.672322 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-podres\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.672799 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.672450 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-sys\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.672799 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.672598 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-lib-modules\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.672799 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.672640 4310 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg5ss\" (UniqueName: \"kubernetes.io/projected/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-kube-api-access-rg5ss\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.773159 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.773124 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-proc\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.773339 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.773171 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-podres\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.773339 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.773218 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-sys\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.773339 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.773256 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-proc\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.773339 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.773315 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-sys\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.773339 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.773335 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-lib-modules\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.773573 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.773348 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-podres\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.773573 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.773362 4310 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg5ss\" (UniqueName: \"kubernetes.io/projected/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-kube-api-access-rg5ss\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.773573 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.773420 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-lib-modules\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.783872 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.783843 4310 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg5ss\" (UniqueName: \"kubernetes.io/projected/48138ef4-98da-4eaa-99e2-e9fb5af9bf9d-kube-api-access-rg5ss\") pod \"perf-node-gather-daemonset-8ctkq\" (UID: \"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d\") " pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:48.924679 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:48.923256 4310 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:49.211060 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:49.210504 4310 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-586q5_5ae0a868-357a-4fe8-a3d9-ea6aff6c4e13/konnectivity-agent/0.log" limit=1024 Nov 02 18:20:49.237496 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:49.237467 4310 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq"] Nov 02 18:20:49.240736 ip-10-0-135-9 kubenswrapper[4310]: W1102 18:20:49.240698 4310 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod48138ef4_98da_4eaa_99e2_e9fb5af9bf9d.slice/crio-293c980ef986247ec9fea5fbd9bcb68da058af3c1dd2dad2c3824e4532719f4f WatchSource:0}: Error finding container 293c980ef986247ec9fea5fbd9bcb68da058af3c1dd2dad2c3824e4532719f4f: Status 404 returned error can't find the container with id 293c980ef986247ec9fea5fbd9bcb68da058af3c1dd2dad2c3824e4532719f4f Nov 02 18:20:49.553101 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:49.543878 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" event={"ID":"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d","Type":"ContainerStarted","Data":"32ec9918c891264847fd23870e50d1e89c3021a69fd091877a24c4758ad022ff"} Nov 02 18:20:49.553101 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:49.543943 4310 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" event={"ID":"48138ef4-98da-4eaa-99e2-e9fb5af9bf9d","Type":"ContainerStarted","Data":"293c980ef986247ec9fea5fbd9bcb68da058af3c1dd2dad2c3824e4532719f4f"} Nov 02 18:20:49.553101 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:49.544798 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:49.563821 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:49.563773 4310 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" podStartSLOduration=1.5637570790000002 podStartE2EDuration="1.563757079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 18:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:49.561689123 +0000 UTC m=+2447.004771762" watchObservedRunningTime="2026-11-02 18:20:49.563757079 +0000 UTC m=+2447.006839714" Nov 02 18:20:50.926500 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:50.926462 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b68b9d69f-jlqkc_24c27b5e-11b0-489c-b3cd-6eface16a294/console/0.log" Nov 02 18:20:50.961482 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:50.961453 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-57wqv_a34b1f69-0317-462a-981b-51ae91f9e8b5/download-server/0.log" Nov 02 18:20:52.791208 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:52.791178 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tz2kg_f27a1622-72d7-411f-baba-028b6c2f94ee/dns/0.log" Nov 02 18:20:52.812906 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:52.812871 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tz2kg_f27a1622-72d7-411f-baba-028b6c2f94ee/kube-rbac-proxy/0.log" Nov 02 18:20:52.894828 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:52.894798 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-59bfz_4307ecfe-be36-4891-9c9d-4c649e9aca70/dns-node-resolver/0.log" Nov 02 18:20:53.531874 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:53.531840 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c84fd649c-lxh8c_2af0994d-e5fb-4ef5-af83-3662154a29fd/registry/0.log" Nov 02 18:20:53.593638 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:53.593605 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2crqk_a3b7e801-25bc-4f6e-be57-f3131d2ce39c/node-ca/0.log" Nov 02 18:20:56.570112 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:56.570062 4310 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4fmsl/perf-node-gather-daemonset-8ctkq" Nov 02 18:20:57.760105 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:57.754505 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8f8hh_26d61838-6558-489f-a369-b8dd6afe14f5/discovery/0.log" Nov 02 18:20:57.785900 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:57.785795 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555d4cc554-ssg7x_4de65c0e-62d2-4e29-b13f-309a0ce43deb/kube-auth-proxy/0.log" Nov 02 18:20:57.940781 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:57.940730 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8ftbd_8d9f41e4-d435-4f1f-95dc-858820a331bf/istio-proxy/0.log" Nov 02 18:20:58.035537 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:58.035456 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-l98w5_d1be36c1-1e0e-478d-bdeb-7a9704cee2e0/istio-proxy/0.log" Nov 02 18:20:58.137067 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:58.137032 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-vkwgh_1a783ce1-3916-4723-a400-9155ea747d15/istio-proxy/0.log" Nov 02 18:20:59.896766 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:20:59.896729 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7r6cv_486c0825-8a27-4863-bbb7-27e11440fb83/serve-healthcheck-canary/0.log" Nov 02 18:21:00.812421 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:00.812380 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zg6hp_5350eca9-94c7-4dc9-a571-ec16e055d94d/kube-rbac-proxy/0.log" Nov 02 18:21:00.838653 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:00.838620 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zg6hp_5350eca9-94c7-4dc9-a571-ec16e055d94d/exporter/0.log" Nov 02 18:21:00.868592 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:00.868557 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zg6hp_5350eca9-94c7-4dc9-a571-ec16e055d94d/extractor/0.log" Nov 02 18:21:02.827384 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:02.827355 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8f8hh_26d61838-6558-489f-a369-b8dd6afe14f5/discovery/0.log" Nov 02 18:21:02.837265 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:02.837239 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555d4cc554-ssg7x_4de65c0e-62d2-4e29-b13f-309a0ce43deb/kube-auth-proxy/0.log" Nov 02 18:21:02.927365 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:02.927299 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8ftbd_8d9f41e4-d435-4f1f-95dc-858820a331bf/istio-proxy/0.log" Nov 02 18:21:02.980858 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:02.980795 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-l98w5_d1be36c1-1e0e-478d-bdeb-7a9704cee2e0/istio-proxy/0.log" Nov 02 18:21:03.039255 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:03.039224 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-vkwgh_1a783ce1-3916-4723-a400-9155ea747d15/istio-proxy/0.log" Nov 02 18:21:03.529629 ip-10-0-135-9 kubenswrapper[4310]: E1102 18:21:03.529556 4310 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zg6hp" containerName="extractor" Nov 02 18:21:03.810919 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:03.810833 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw_8329f1cc-0f29-4f30-9597-7a50d819d14a/manager/0.log" Nov 02 18:21:03.830550 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:03.830523 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6f4dd5dc4d-r6dtw_8329f1cc-0f29-4f30-9597-7a50d819d14a/copy-manifests/0.log" Nov 02 18:21:05.784843 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:05.784816 4310 scope.go:122] "RemoveContainer" containerID="d29c273fee008e503b84c456429aa2cf44c962675addfdae1742420fdccbec92" Nov 02 18:21:05.790688 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:05.790658 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-fdd7ff9-r2xj6_325e80f9-6751-44b2-a3b2-0c2b1d2e4def/manager/0.log" Nov 02 18:21:05.828890 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:05.827142 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4pwjv_05242e7d-02d8-4b62-a200-cdd9f108e64a/manager/0.log" Nov 02 18:21:05.919363 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:05.919310 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rxjvx_af9c459a-3726-41aa-b94d-02bdcfd9d10e/manager/0.log" Nov 02 18:21:06.372139 ip-10-0-135-9 kubenswrapper[4310]: I1102 18:21:06.372106 4310 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-698c564c8b-rgrcd_6058e7f5-08e4-47e6-a656-9fc94d877250/manager/0.log"